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ABSTRACT 



An investigation is made of the Navy Exploratory Develop- 
ment Program; its newly revamped management system, its 
organization and underlying rationale. The investigation 
then addresses the processes of program selection, review and 
evaluation as applied to technology base programs, followed 
by an overview of a methodology called "Evaluation Research" 
as it applies to goal oriented programs. The paper is dir- 
ected toward determining the effectiveness of technology base 
programs, with a major emphasis on the changing structure of 
the Exploratory Development Program within the Department of 
Defense . 
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I. 



INTRODUCTION 



There has never been a period in history in which the 
determination of the worth of programs within the Department 
of Defense (DoD) structural framework has been so critically 
viewed. Adding to this observation is the discernible fact 
that relative budgets within the Research and Development 
(R&D) Program are steadily decreasing. 

This concern encompasses total system acquisition, oper- 
ations and maintenance, hardware and softxi^are development, 
concept feasibility, pure research and all other programs 
intermediate. Further, the concern is broad-based to the 
extent that it has become a prime distress not only to the 
government sector of DoD, but also to the supporting indus- 
trial counterparts. 

Owing to the desire then to better justify on-going pro- 
grams, to promote innovation and concentration on highest 
leverage opportunities, to expedite the transfer of technolgy 
and projects to advanced development (category 6.3) and use 
of new technical capabilities developed in research (category 
6.1) and to demonstrate relevance and cost effectiveness, 
the Research and Development program in total is currently 
undergoing a modification of the existing management planning 
system, particularly the Exploratory Development Program (EDP) . 

The objective of this paper then is really twofold; 

(1) to investigate the changing structure, organization and 
underlying rationale of the Exploratory Development Program 
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in the Navy and (2) to investigate the processes of program 
selection, review and evaluation as applied to technology 
programs within the EDP. Included will be a brief on the 
development of the EDP, a treatise on why it is a necessary 
and contributing element of R&D and also a discussion of what 
the future might hold for determining the "worth" of technol- 
ogy type programs . 

Because of the significance placed on an understanding 
of the operations of the EDP as a necessary and crucial pre- 
requisite to developing a meaningful evaluation methodology, 
the majority of this paper is focused on identifying the 
foundations and workings of the newly revamped Exploratory 
Development Program. This understanding is essential before 
any succinct attempt can be made to alleviate the discrepan- 
cies in the review and evaluation techniques, existing today. 
Hopefully, a closer interpretation of the EDP will facilitate 
a more viable methodology for determining the Return-on-Invest- 
ment (ROI) of future technology programs within DoD. 
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II. BACKGROUND 



A. THE TECHNOLOGY BASE 

Somewhere between basic Naval research on one end of the 
Research and Development (R&D) spectrum and full scale devel- 
opment on the other, fall a great many projects which come 
under the heading of applied technology, or technology base 
programs. The technology base is the total reservoir of 
organized knowledge from directly and indirectly sponsored 
basic research into physical and social phenomena and the 
feasibility of new processes, techniques and components for 
using them (Ref. 1) . Its end result is new organized knowl- 
edge. The base also retains fallout information and contri- 
butions left over from exploration and development of specific 
systems. The creation and exploration of candidate systems, 
in turn, is shaped by the information available from the tech- 
nology base. 

One of the largest and perhaps most dynamic businesses in 
the United States today is that dealing with the acquisition 
of major weapon systems for the Department of Defense. Due to 
the size and complexity of the federal government and its 
associated agencies, management of the affairs of this organi- 
zation requires a tremendous amount of activity and coordination. 

This business entity, not unlike any other, maintains a 
growth posture made possible by both internal and external 
influences. These influences are visible through the technol- 
ogy development programs supported by the organization. Thus, 
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a most important faction of the R&D community within the 
Department of Defense (DoD) is that responsible for developing 
and maintaining a strong technology base. 

The technology base is supported by both Government and 
private activities. Private groups (such as industrial com- 
panies) and nonprofit organizations (such as Federal contract 
research centers and universities) both contribute to the 
technology base and recommend technology base programs, just 
as do the federal laboratories and other supporting agencies. 

Within DoD, the defense laboratories are charged with main- 
taining the technology base by sponsoring activities in support 
of potential systems. Laboratory research objectives are 
planned to generate better technological tools five to twenty 
years in the future. There does indeed exist a requirement 
for the technology to be related to mission/operational needs. 

Due to the seemingly nebulous nature of these technology 
oriented programs, a great deal of scrutiny has been, and con- 
tinues to be, given them, with primary emphasis of the concern 
aimed at their resultant value. Because of the presence of 
intangible outputs in the sense of quantification, coupled 
with an evidence of decreasing relative budgets, technology 
administrators at all levels within the command have been, and 
continue to be, faced with the dilemma of determining which 
programs should receive their attention. Their problem is 
essentially twofold; (1) How are they to determine which of 
the many proposed (alternative) programs should be selected 
for support, and (2) how are they to evaluate the effectiveness 
of their programs? 
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It is very difficult to know how much should be spent for 
technology base activities because payoffs are virtually 

unpredictable and only rarely are they realized at some future 
date in a directly traceable manner (Ref. 1) . Operational 
needs can be met most effectively at the lowest possible cost 
by innovative products built on new technology being created 
for its own sake (pure technology) as well as that which 
remains as fallout from other system efforts (applied 
technology) . 

A significant problem in defense has been to control the 
relationship between technology base activities and system 
acquisition programs. Technology base projects have gone on 
to develop more than just the raw elements and material to 
meet as yet unexpressed needs. Traditionally they have devel- 
oped subsystems and system concepts independently of specific 
needs. Once developed, subsystems have sought out homes in 
new system candidates and systems in turn have searched for a 
need they could satisfy. Further, these subsystems have been 
old technologies extended to improve the performance of old 
kinds of products. 

There is a common misconception that defense systems have 
suffered because they have embodied too much "new technology" 
and, as a result, have become too complex, sophisticated and 
expensive (Ref. 1) . The explanation for this is that what is 
"new" about "new technology" is the higher level of perform- 
ance demanded from the old technical approaches. In fact. 
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subjective ratings have indicated that systems, while becoming 
more complicated and expensive, have become less technologi- 
cally advanced. 

Applying new technology to meet more stringent needs can, 
in fact, result in simpler, less costly systems. As a common 
example, the transistor development has markedly reduced the 
cost and size of electronic systems and improved their reli- 
ability, maintainability and other cost of ownership factors. 

The need for an active and viable technology base is thus 
extremely important. The larger the span of technical and 
scientific activity sponsored by the Government, the more 
extensive will be the optional technological choices avail- 
able to the system designer. It is imperative that cost- 
benefits be maximized in providing the technology base for 
new systems. This then can only be realized when the base 
program is driven from operational needs and mission require- 
ments. This, in turn, requires a most judicial and expedient 
methodology for determining the criteria for the selection 
and evaluation and the effectiveness of the technology base 
programs in the R&D environment. 

Just as critical as a well defined and operable methodology 
for determining selection and evaluation criteria, is the 
understanding of the actual program, or system, which is to 
be operated on. No attempt could ever be made to determine a 
program's worth, or return on investment (ROI) , without a prior 
and thorough understanding of the functions and underlying 
philosophies of that program. It is crucial that a sound 
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foundation be established from which goals and strategies 
can be derived rather than the traditional dollar value of 
output . 

B. THE EXPLORATORY DEVELOPMENT PROGRAM AS THE CRITICAL LINK 
While many definitions of exploratory development exist, 
there are several functions that should be fulfilled by this 
category of Research and Development. These might be grouped 
into the following areas; 

1. The translation and application of research results 
to practical use, including the formulation of new 
technology areas. 

2. Feasibility demonstrations of component and system 
hardware concepts. 

3. Applications of new technology to existing components 
or systems. 

The nature of the preceding functions implies that both the 
inputs and outputs of exploratory development are quite 
varied, and can be thought of as a continuum. Maximum benefit 
in an Exploratory Development organization would appear to 
accrue from having the continuum functions in close proximity. 

The purpose of the Exploratory Development program as 
delineated by NAVMATINST 3910. 7B of 3 October 1974 is as 
follows : 

The purpose of the Exploratory Development program 
is to develop the technological wherewithal! to solve 
specific Navy and Marine Corps problems, and to identify 
and exploit new technological opportunities which may 
stimulate new operational concepts within the operating 
forces. It is the policy of the Chief of Naval Develop- 
ment that the program include concept formulation effort 
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in the form of analytical and experimental effort to 
help identify problems, determine alternative solu- 
tions thereto in terms of prospective systems, sub- 
subsystems and techniques, and demonstrate the technical 
feasibility of those solutions to a degree which 
warrants their consideration for support under Advanced 
Development. Selectively, where a system or subsystem 
is relatively small and of low technical risk, the 
program may support effort aimed at acquiring informa- 
tion prerequisite to commencement of Engineering 
Development. Such effort, however, should stop short 
of engineering refinements characteristic of proto- 
typing development for procurement, inventory or opera- 
tional deployment. While the program may also encompass 
analytical and experimental work on the technologies 
directly related to materials, components, processes, 
techniques , and individual equipments of Navy and 
Marine Corps interest, broad-based investigations in 
physical, biological, environmental, medical, behavioral, 
social and engineering sciences characteristics of the 
Naval Research program are not permitted. Effort of 
an exploratory nature, which has a primary purpose of 
direct support of specific Advanced, Engineering or 
Operational Systems Development projects should be 
funded and accomplished as an integral part of those 
projects, subject to close coordination with related 
effort in the Exploratory Development program. 

The operation of this system has several characteristics 

that are significant in the management of applied technology 

(category 6.2) programs (Ref. 2). 

• Very broad technological base requiring judgements and 
technical expertise in many fields at all levels of control. 

• Many layers of control--DDR&E , ASN (R&D) , CND, SYSCOMs, 
Offices, DLP, DNT, laboratories — and necessity for coordina- 
tion of tasks with related programs as well as with pure 
research (category 6.1) and advanced development (category 
6.3) . 



• Coordination necessary between services and with the 
industrial complex. 

• Results of program efforts are seldom visible or recog- 
nized in the system that reaches operational use. 

• Categorization of tasks is not easily relatable to naval 
responsibilities, missions, operational concepts, or systems. 
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• CNO's guidance on Naval needs is not directly trans- 
latable into R&D requirements. CNM does not interpret oper- 
ational needs for SYSCOMs. Each SYSCOM (and suborganization 
within SYSCOM) employs a different subjective means in 
defining R&D goals. There is no priority assigned to indi- 
vidual tasks. 

• The system creates a large number of low level funded 
tasks (approximately 650 Task Area Plans (TAPs) (DD1634) 
yearly) . 

• Approved TAPs can be underfunded or not funded at the 
discretion of SYSCOMs or laboratories without reference to 
CND or DNT. 

• Reductions in funding are frequently made by propor- 
tionate cuts which results in perpetuation of sub-critical 
programs and reduced effort on critical programs. 

• Excessive effort is devoted to making minor changes in 
funding levels with resulting turbulence. 

• Inordinate effort is devoted to management, funding 
changes and justifications of programs which takes technical 
personnel away from productive tasks. 

• The system fosters "micro-management." Program reviews 
frequently do not result in "make-a-dif f erence" changes. 

• Protection of in-house effort emphasized as contrasted 
to contractor effort. 

• Delay in exploiting available technology. 

• Overemphasis on evolutionary and fix-it projects. 

• Inadequate scientific and technical advice in special- 
ized fields involving programs of several organizations. 

• Critical problems exist related to coupling the 6.1, 

6.2, and 6.3 programs to initiate an Engineering Development 
(6.4) program. 

It thus becomes evident that a crucial prerequisite to 
any understanding of the methodologies of Exploratory Develop- 
ment program selection and evaluation is a thorough under- 
standing of the program itself. For this reason, and owing 
to the fact that EDP is at this time undergoing major organi- 
zational and structural revision, a significant amount of 
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attention is shown this subject in this thesis. The EDP is 
indeed the "critical link" in terms of developing major weapon 
systems within the Department of Defense, and understanding 
the functions and operations of it are also critical in terms 
of measuring program worth. The changes in the program have 
been initiated in an effort to alleviate the negative aspects 
of the previously mentioned characteristics of the program 
operation. It is felt that only through an understanding of 
the function/operational aspects of the EDP can the concept 
of Return-on- Investment be addressed, since this must ulti- 
mately be linked to the organizations' goals and strategies 
rather than dollar value of output. 
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III. EXPLORATORY DEVELOPMENT PROGRAM 



A. A RATIONALE FOR A POLICY PLATFORM 

Because of the tremendous size and complexity of the 
Federal Government and its many associated agencies, manage- 
ment of the affairs of this organization constitutes a very 
significant amount of activity and coordination. Accordingly, 
a great deal of scrutiny is given the policy platforms and 
expenditure characteristics within sub-organizations of this 
complex. 

This section will investigate the Exploratory Development 
Program within the Department of the Navy, examine the policy 
of the EDP and the rationale behind that policy , examine 
several influences which are evidenced by the technology pro- 
graim, and provide a personal critique of that policy plat- 
form currently existing for the EDP from the standpoint of a 
public procurement. 

It is felt that a comfortable understanding of the EDP 
policy is a valuable and key element which allows a better 
comprehension of the significance of the EDP structural 
change, thus leading toward a discussion of the present pro- 
gram organization and characteristics. The approach then 
will be to first address the broader issue of external policy 
influences and then focus on particular program policies as 
delineated by the Assistant Secretary of the Navy for Research 
and Development (ASN CR&D) ) . 
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1 . The EDP as a Public Program 



If any government program could be labeled "public," 
the Exploratory Development Program must certainly be con- 
sidered as a leading contender. The business of this program 
is certainly conducted for the general welfare of the people 
of the United States through the guise of building and main- 
taining a strong and viable defense posture. Clearly then, 
the output of this entity is to be labeled a public good, 
since it is consumed by all members of the community of the 
United States . The control of expenditures is governed by 
the internal processes within the Department of Defense (DoD) , 
since the costs of production cannot be maintained by competi- 
tive pricing. 

Because of the nature of the product in the Explora- 
tory Development Program, the citizens do not actively par- 
take in setting policy or in providing an oversight function. 
This task is left to entrusted officials within DoD, as well 
as the mix of activities performed. The financing aspect of 
the EDP comes under close scrutiny through the auspices of 
both Congress and the Office of Management and Budget (OJIB) 
in the Executive Branch of government. Programs on-going 
must continually be justified and reviewed for content, return, 
applicability and need. 

There is a facet of the EDP which presents itself as 
an externality to the private sector. This is the area which 
is embodied by the notion of "technology transfer." This 
term is used to imply that technology, as the output from a 
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research effort, can be utilized productively in many loca- 
tions and organizations over and beyond its initial specific 
mission (Ref. 3) . The process of technology transfer in the 
EDP has traditionally been one of primarily outflux from DoD. 

The working policy and structural framework of the 
EDP is built to closely resemble a standard "rational model" 
organization (Ref. 4) . This will become quite evident later 
when the working policy/critique is presented. However, 
implementation of this policy is probably more closely repre- 
sentative of the classical "bureaucratic model," with only 
a moderate trend toward a politically contrived system (Ref. 

5) . 

It should be submitted that no "Operational Code" 
exists which might be any more distinctive than the academic 
models just mentioned. This hypothesis is supported by 
noticing the well defined organizational concepts and general 
propositions under which the EDP functions; high degree of 
standardized procedures, uncertainty avoidance, centralized 
policy formulation, decisions made by leaders, presence of 
parochial priorities, actions are organizationalized , change 
is definitely incremental, long range planning is prevalent, 
goals and tradeoffs are of primary concern, decisions are 
based on administrative feasibility, change is directed from 
higher authority and there is limited flexibility in the line 
functions of the organization. Additionally, one can very 
obviously witness well defined rules of the game, actions 
which come about as a result of political inducements, definite 
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separation and identification of chiefs and indians and much 
mis-communication , all of which point toward a strong influ- 
ence of the political model characteristics (Ref. 4) . 

Having provided an overview of the structure of the 
EDP and its operational characteristics and piiblic attributes, 
attention now will focus more closely on the particular policy 
of the program on which its platform is built. Any discus- 
sion of the Congressional and Presidential influences on 
the EDP has been negated because it is not felt to be a pro- 
gram, primarily because of its size and technical complexity, 
which lends itself to the normal control imposed by those 
forces. All interactions which do exist are enacted at the 
ASN (R&D) level, the results of which follow. 

2 . EDP Rationale 

A review of the Navy's Exploratory Development Pro- 
gram by the Assistant Secretary of the Navy (R&D) led to the 
recent concern to re-establish the policies which are intended 
to govern that section of Research and Development. As a 
result, the ASN (R&D) has specified the policies, performance 
objectives, and responsibilities concerned with the overall 
management of the Department of the Navy Exploratory Develop- 
ment Program (Ref. 6). 

It is the belief of the ASN (R&D) that improvements 
are possible in technical strategy, responsiveness to innova- 
tion, transferring results into useful applications, visi- 
bility to management, and demonstration of the rationale and 
cost effectiveness of the Exploratory Development Program and 
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its elements. The underlying rationale of the Exploratory 
Development Program is derived from the fact that this 
activity, like all others, must compete for its resources. 
This implies that the programs be continually justified in 
meaningful and convincing terms . 

According to the ASN (R&D) , to assure that the long 
term benefits are clear. Exploratory Development must demon- 
strate how it (Ref. 6): 

a. Provides technical opportunities which stimulate 
the development of siabstantially improved and/ 
or totally new operational capabilities. 

b. Provides solutions to problems arising in the 
developmental process. 

c. Provides opportunities to regain operational 
superiority or to exploit and enhance an 
existing operational superiority. 

d. Provides for the transformation of new research 
discoveries into useful applications, and for 
the innovative extension of existing technologies 
to significantly more advanced applications. 

3 . Policy Formulation 



Using as a foundation the principles implicit in the 
current operation of the Program, policies have been identi- 
fied which are intended; to enhance the productivity of 
Exploratory Development; and to improve the ability to demon- 
strate the need for, and benefits resulting from, the pro- 
gram's activities (Ref. 6) . 

a. The Exploratory Development Program shall be 
based on a set of dynamic technical strategies 
designed to; 

Cl) take maximum advantage of new technical 
opportunities 

(2) exploit deficiencies in opposition capa- 
bilities, and 
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(3) provide prompt response to perceived 

requirements for superior naval capability. 

b. The Exploratory Development Program shall be 
made visible through an information system per- 
mitting technical, as well as financial audit 
on a current basis to cognizant levels of 
management. The principal intent of the inform- 
ation system is to permit control of the tech- 
nical strategy at higher levels, facilitate 
execution of the programs at the most techni- 
cally competent levels, and facilitate the 
transfer of technology to advanced development. 

c. The Exploratory Development Program shall include, 
to the maximum extent practicable, 'block 
programs' which are identified with major tech- 
nical strategies and under the management of an 
organization whose mission/product line or 
technical competence is most appropriate. 

d. The Exploratory Development Program plan shall 
be consistent with overall assigned ratios of 
in-house to contract work for balanced utiliza- 
tion of the national technological base. 

e. The Exploratory Development Program and its 
elements shall be based on a more explicit 
application of the return-on- investment (ROI) 
concept. 

f. The Exploratory Development Program shall be 
based on cost-effective principles and shall 
consider, as well as budgetary implications, 

the potential costs of acquisition and ownership. 

g. The Exploratory Development Program shall be 
based on providing adequate funding for effective 
exploitation of highest-leverage opportunities. 
When over-all resources are inadequate or 
reduced, funds required for support of such 
projects shall be made available through can- 
cellation or curtailment of projects with lower 
promise of military worth within the framework 

of technical strategy. 

4 . An Assessment 

In general, the rationale and the policies identified 
are not inconsistent with the overall policies of Fiscal and 
Life Cycles of Defense Systems. That is to say, the basic 
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questions of need, technical feasibility and funding avail- 
ability are inherent in the policy information for the 
Exploratory Development Program. However, an immediate con- 
cern which comes to mind is whether the policies, being 
quite broad and abstract by their very nature, can be sup- 
ported and/or even interpreted at the working levels. This 
concern is not peculiar to this particular policy formula- 
tion, rather it deals with the structure of policy enactment 
through any large organization. 

Because the policies have already been identified, 
the approach that will be taken now will be to address each 
policy statement separately and provide personal comment as 
deemed appropriate. 

Policy A: Primarily the emphasis should be placed 

on the term technical strategy, as a means of assuring that 
the programs supported are productive, needed and will pro- 
vide benefit. It would appear that a sophisticated and 
well-managed communication (information flow) scheme is 
necessary to carry out the desires of top management. Worth 
questioning is the productivity, however, of focusing atten- 
tion on the deficiencies of the opposition to the detriment 
of encouraging an offensive approach. Further, to provide 
for 'prompt response' would necessarily imply that the cor- 
porate memory and developmental capability of the in-house 
sector be maintained. 

Policy B: Almost without exception would there be 

an agreement with the notion of a needed buildup of a more 
meaningful information system. It is always potentially 
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possible though that a problem of unidirectional information 
flow will ultimately come into being. There also appears to 
be some conflict with the idea of maintaining control at the 
higher levels, and at the same time encouraging decentraliza- 
tion of the EDP. Decentralization is essential in order to 
promote and produce technical excellence and thus realize 
maximum productivity. 

Policy C: There currently exists much support of the 

notion of 'block programming' on the basis of less duplica- 
tion of efforts, greater knowledge of state-of-the-art in the 
particular product lines, and increased ability to select and 
evaluate the individual programs within the EDP . 

Policy D; Clearly the bottom line here is to bolster 
and maintain a strong technology base, whether that be pro- 
vided for by industry or in-house. An obvious question which 
surfaces given a predetermined ratio, is just where in the 
organization that ratio must be measured — for this will have 
a definite influence on accountability within the system. It 
should be realized that the ultimate decision should be one 
which yields the maximum advantage to the government's carry- 
ing out its mission. This philosophy necessarily requires a 
consensus that those goals and missions have properly and 
adequately been delineated. 

Policy E; While the notion of return-on-investment 
appears relatively straightforward at first glance, its appli- 
cation in an area such as Exploratory Development is extremely 
difficult to enact. This is due to the obvious lack of 
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tangible, and thus quantifiable, output from the individual 
programs. However, in the context of wanting to maximize 
the benefits of the dollars invested, the concept is crucially 
appropriate. 

Policy F: As much as is possible, attention should 

be given to the potential downstream costs of acquisition 
and ownership. It is this basis which will ultimately provide 
for a rational decision as to which programs will be supported 
in future Exploratory Development programs. 

Policy G; Certainly the philosophy most often 
followed in practice would lead one to select those alterna- 
tives yielding the greatest leverage. Seriously questionable, 
however, is that portion of this policy that suggests each 
program be evaluated on its own merits and would be subject 
to cancellation if it showed less favor in terms of military 
work. It is of general belief (supporting that belief exis- 
ting at the NAVMAT level) that in most cases, existing pro- 
grams will continue to be funded unless there is obvious 
reason to alter that decision on the basis of either lessened 
or increased need. 

5 . Concluding Comments 

There is a general trend in the Navy's Research and 
Development Program, including the Exploratory Development 
Program, to shift back to an originally supported organiza- 
tional structure of being decentralized. This is due to a 
large extent to the desire to promote accountability at all 
levels of the R&D organization. Additionally, it is observed 
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that trends definitely indicate future total budgets to rise, 
while the percentage allocation to the Exploratory Develop- 
ment category will, as has been experienced in the past 
decade, definitely decline . As a result of this observation, 
it is noted that technology development is definitely not 
independent of weapon system development. 

It is noteworthy in summary to exploit the under- 
lying principles evidenced in the policy formulation for the 
Exploratory Development Program; 

• Supportive of technological advances of funda- 
mental importance to operational needs. 

• Quest for technical and managerial excellence at 
all levels . 

• Recognition of the necessity for both top-down 
direction and bottom-up inputs . 

• Accountability essential at all executive levels. 

• Necessity for strategies as tools to implement 
policies . 

• Recognition that sometimes technical advance must 
be injected into the mainstream of the system 
forcibly . 

• Recognition of essential need for adequate 
information systems to allow interaction . 

While it is a rather straightforward task to outline 
a rational approach, it is extremely difficult, if not 
impossible, to encounter a purely rational organization in 
operation. Influences, personalities, and constraints are 
entirely too numerous to allow anything other than some form 
of an "individualized" organization to exist today, whether 
that organization be public, private or some mix of the two 
extremes . 
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An investigation of the EDP policy does, however, 
serves to provide the foundation on which its organization 
must be structured. Further, this foundation (rationale 
and policy) provides the impetus behind the current EDP mod- 
ification, which is the subject matter of the following 
section of this thesis. It is important that the reader be 
aware that in addition to gaining a better understanding of 
the "real” system in operation today, the material presented 
will hopefully provide a context in which to review the sub- 
jects of program selection and evaluation in applied tech- 
nology in a more meaningful manner. 

B. STRUCTURE OF EDP (A CHRONICLE ) 

The creation of the offices of the Director, Defense 
Research and Engineering (DDR&E) , Assistant Secretary of the 
Navy (Research and Development) , and Deputy Chief of Naval 
Operations (Development) (DCNO(D)) back in 1959 set the stage 
for some revolutionary changes in the way R&D programs , and 
hence Exploratory Development programs, would be planned, 
justified and implemented for the next two decades. The 
concept of a complete R&D spectrum with basic research on the 
one end and full scale development on the other was inherent 
in the procedures of the first decade. 

Technology programs were established which covered a wide 
variety of efforts, ranging from pure research in components 
and subsystems to fairly sophisticated experimental hardware. 
These technology programs at that time were characterized as 
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being comprised of projects more closely focused on end- 
products than research but entailing a higher level of uncer- 
tainty and lower financial commitment than full scale 
development. 

Bureaus and offices were established in areas relevant to 
their assigned responsibilities. Project listings were 
required to be structured according to the format of Table I 
(Ref. 7) . So for the first time, senior staffs in the Depart 
ment of the Navy and the Office of the Secretary of Defense 
had a viable framework for scrutinizing the exploratory devel 
opment effort and the funds allocated to it. 



Table I Exploratory Development 
Functional Areas, 1962 



FOOl 


Target Surveillance 


F002 


Navigation 


F003 


Environmental Surveillance 


F004 


Integrated Surveillance 


F005 


Command Control 


F006 


Communications 


F007 


Data Processing 


F008 


Weapons and Ordnance 


F009 


Guided Missiles 


FOlO 


Jamming and Deception 


FOll 


Naval Defense Applications 


F012 


Aircraft and Aircraft Support 


F013 


Ships and Submarines 


F014 


Boats and Amphibious Vehicles 


F015 


Logistics 


F016 


Personnel Administration 


F017 


Training 



The process of planning and justifying programs in the 
early 1960s is broadly depicted in Figure 1 (Ref. 7) . There 
was a trend in the early years for the bureaus and offices to 
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R&D Program Planning and Justification Process, 

Early 1960's 
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employ a "carte blanche" attitude to initiate new programs 
in response to general requirements. This philosophy began 
to erode, however, with the enactment of PPBS , and eventually 
the organization became extremely controlled, and hence, cen- 
tralized by the end of the first decade. 

In 1970, DDR&E then introduced two new types of documents 
which more clearly stressed the concern of program worth, 
justification and relationship to mission areas. They were 
called Technology Coordinating Papers and Area Coordinating 
Papers. They were intended to highlight program gaps, over- 
laps, and need for improved coordination among the Services 
(Ref. 8) . 

And so, during the early baseline period, technology pro- 
grams represented a segment of the Navy's R&D program in 
which the management operated with little or no outside inter- 
ference. Guided only by the most general requirements, bureau 
personnel relied heavily on Fleet and laboratory personnel, 
discussions with the Fleet and experience of their own offi- 
cers and engineers to identify problems, initiate programs 
and evaluate their on-going projects. 

There was a trend toward greater emphasis and dependence 
on the Navy laboratories, i.e., block programming. The bureau/ 
SYSCOM point of view initially was to oppose block programming 
on the grounds that it amounted to delegating authority to 
the laboratories without commensurate accountability for ful- 
filling SYSCOM responsibilities. The idea of block programming 
has since become a reality, and is now in full effect, with 
nearly full support, in the Navy organization. 
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Block programming/ as interpreted today, is a means for 
providing technical management in the laboratories and Systems 
Commands with the necessary authority (including control of 
funds) to plan and manage technical effort related to the 
pursuit of significant objectives. As will be seen later, 
block programming is a significant influence in the current 
planning and justification process proposed for the Explora- 
tory Development Program. 

C. A CHANGING CONTEMPORARY PROGRAM 

The Navy Exploratory Development Program is just nearly 
completing a transition from a system, which has been in 
operation for a number of years and is relatively well known 
by the Navy R&D community, to a new system. Central to these 
modifications is the use of dynamic "technical strategies," 
designed to focus exploratory development more directly and 
visibly on Department of the Navy operational needs. The 
first implementation of the program utilizing these technical 
strategies will commence during the current fiscal year. 

This program reorganization is a result of the CND/DCNM(D) 
having been directed by the ASN (R&D) to establish and imple- 
ment a program to improve the exploratory development system, 
to improve higher echelons' understanding of the system and 
to justify 6.2 expenditures. 

As indicated previously, under the former 6.2 management 
system the program consisted of a large number of low level 
funded tasks which were negotiated primarily between staff 
personnel of the SYSCOMs and the laboratories without 
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direction of either top level management or the best qualified 
technical personnel who are generally located at the labora- 
tories . The relationship between funded tasks and operational 
needs was not altogether visible and the contributions that 
new technologies could make to increase effectiveness or 
decrease costs were not apparent. As a result, justification 
of the program to Congress, OSD and to ASN (R&D) was not con- 
vincing. There was a lack of funding focus on high payoff 
innovation and on feasibility demonstration of critical end 
item products. Further, there had been an inadequate inter- 
action of 6.1 with 6.2 and coupling with 6.3 and a lack of 
coordination with related work by other services and industry. 

The objectives of the modified 6.2 management system were 
then identified to (Ref. 9) ; 

• Provide top-down direction by the use of technical 
strategies, prioritized technical objectives and 
definition of major program thrusts designed to meet 
operational requirements. 

• Provide visibility by relating the program to Navy 
functions and to operational needs . 

• Promote innovation and concentration on highest 
leverage opportunities. 

• Expedite transfer of projects to advanced development 
(6.3) and use of new technical capabilities developed 
in research (6.1). 

• Demonstrate relevance and cost effectiveness. 

These objectives were to be achieved by clear definition 
of organizational responsibilities, development and execution 
of programs to achieve approved technical objectives, allo- 
cation of blocks of funds to organizations and delegation of 
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authority to implement programs to meet assigned responsibili 
ties. CND is the designated responsible entity to ASN (R&D) 
for operation of the system. Appendix A is included to delin 
eate the EDP management responsibilities, including design 
principles and roles of the various organizations in the 6.2 
Program. 

There are four basic elements of the new 6.2 planning sys 
tern which utilize the technical strategy concept. 

1 . Direction by Technical Strategy 

Designated strategists identify, define and analyze 
mission needs and on-going or potential technical efforts 
related to satisfying those needs. Mission needs are derived 
from analyses of threat, preferred operational concepts and 
system and subsystem concepts. Current programs and work 
by industry and other services are major factors in assessing 
technical potential. Technical objectives are derived from 
mission needs and potential technology. The strategies then 
are used to define the desired major thrust in prioritizing 
the technical objectives. 

These dynamic technical strategies are designed to 
take maximum advantage of new technological capabilities, 
exploit deficiencies in opposition capabilities and provide 
prompt response to perceived requirements for superior naval 
capabilities (Ref 6). Basically, strategic planning is not 
a new idea, and it implies ”... the process of determining 
the major objectives of an organization, and the policies and 
strategies that will govern the acquisition, use and disposi- 
tion of resources to achieve those objectives” (Ref. 10) . 
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The enactment of an Exploratory Development program 
policy based on the idea of technology strategies is a signif 
icant breakthrough in terms of developing a more meaningful 
rationale for program/project evaluation. The primary reason 
for this is that heretofore, organizations had neglected to 
recognize that there had been a definite lack of clarity in 
just what they were trying to measure. This belief is sup- 
ported by various studies of productivity measurement and pro 
gram evaluations (Ref. 11) . 

It should be recognized that Exploratory Development, 
like every other activity, must compete for its resources. 
Although the necessity for an extensive program is obvious to 
those within the RDT&E establishment, to assure maintenance 
at the desired level, the program must continually be justi- 
fied in meaningful and convincing terms. To assure that long 
term benefits are clear. Exploratory Development thus must 
demonstrate how it: 

a. provides opportunities which stimulate the 
development of improved or totally new 
operational capabilities, 

b. provides solutions to problems arising in 
the development process, 

c. provides opportunities to regain operational 
superiority or enhance existing superiority, 

d. provides for the transformation of new research 
discoveries into useful applications, and the 
extension of existing technologies into signifi- 
cantly more advanced applications. 

To accomplish this, then, the system being assembled 
at the NAVIdAT 03 level addresses not only objectives, strate- 
gies and tactics (OST) , but also interrelates them and 
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describes how the system works to harness research and devel- 
opment for the achievement of definite organizational goals 
(Ref. 12) . 

Essentially the OST system is based on three major 
ideas: (1) formulation of an OST plan consisting of a hier- 

archal arrangement of corporate objectives, technical strate- 
gies, and developmental tactics, (2) the super position of 
this organization over the normal operating organization, 
and (3) the use of line managers in the operating organiza- 
tion as action officers. Figure 2 presents this system in a 
somewhat simplified manner. 

This current model thus contains all the elements of 
the historical and modern approaches, e.g., cost, timeliness, 
risk, weapon systems, value measures, etc., and also provides 
the additional and quite significant notion that the goals 
and policies of the organization need be accounted for. 

The strategies then, as developed for the Navy EDP, 
will indicate allocations recommended for additional objectives 
with justification. DNT will provide guidance regarding over- 
all concepts for functional areas, and also to strategists 
through the inclusion of relevant material from program objec- 
tive memorandums . 

2 . Negotiating the Program 

With the implementation of the new policy, all explor- 
atory development programs will now be prenegotiated between 
SYSCOMs, laboratories and DNT prior to beginning each fiscal 
year. After the programs have been developed they then are 
recommended to DNT for allocation of funds . 
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OST STRUCTURE 




Figure 2 
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There are three types (categories) of programs, the 
characteristics of which determine how the organization will 
manage them. Cat I is broadbased, multi-use technology. It 
is negotiated between DNT and laboratory CO/TDs and funded 
and managed by the laboratories. All "special topic" strategy 
areas are included in Cat I programs. Cat II is concept- 
related effort for which the SYSCOMs are customers. It is 
concerned primarily with exploratory development of systems 
and sxabsystems concepts for accomplishing mission needs. 
Projects for this kind of work are negotiated between the 
using SYSCOMs and the cognizant laboratories. Once negotiated 
and approved by DNT, it is funded directly as a block to the 
performing laboratory which manages it. Cat III work is per- 
formed primarily by industry or other contractors. It in- 
volves primarily work related to transition of concepts to 
advanced development, and it is managed directly by the SYS- 
COMs. Figure 3 shows the "chain of accountability" which now 
exists in the EDP and the types (categories) or programs for 
which the various tiers are responsible. The planning pro- 
cess which has been formulated in consent with the new manage- 
ment system is depicted in Figure 4 . 

3 . Allocation of Funds 

Of the total funds allocated for the EDP, the Chief 
of Naval Development withholds a total of 15 percent of each 
program element to take care of subsequent Congressional cuts, 
to provide for new starts, for developing the strategies, for 
IR/IED, MAT 03 management, the Energy Office and DNT's special 



39 




j STRATEGY 
lADMINISTRATOR 

BUK. PROGRAM 
k MANAGERS 



[STRATEGISTS 



I - BROAD BASED TECHNOLOGY 
n - CONCEPT RELATED TECHNOLOGY 
m - INDUSTRIAL 6.2 TECHNOLOGY 



(DNT $— PERF. LABk 

(MAT 03T $ PERF LA8> 

(SYSCOMS — $— HNDUST. CONCERN) 



FIGURE 3* Chain of Accountability 



40 



FIGURE 4. 



Exploratory Development Program Process Flowchart 
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projects. The bulk of the funds (roughly $2.75B yearly) 
are allocated to SYSCOMs and laboratories. Once the con- 
tracts for work have been negotiated, changes can be made 
only by the SYSCOMs for the CAT III programs and TD of labor- 
atories for CAT I and II programs. Final approval, as before, 
always rests with the Director of Navy Technology. 

4 . Review and Evaluation 

The review and evaluation focuses on two major 
issues: (a) were the resources actually utilized as covered 

in the approved program as amended and (b) to what degree 
were the technical objectives developed by the strategies 
accomplished. It was perceived by management that the basic 
review and evaluation questions should focus on contributions 
to future Navy operations and cost ef f ectivensss . At the 
time of research gathering for this paper there were no de- 
tails available as to how the review would be conducted but 
the objective is clear — substantive evaluation from the view- 
point of Congress, DDR&E and ASN (R&D) and not micromanagement. 
It is anticipated by EDP management that program administra- 
tors, strategists, and a qualified panel of independent 
individuals with knowledge of Navy needs, and other Federal 
and industrial efforts would be involved in review and evalu- 
ation process. However, it should be noted again that no 
clear and definitized methodology yet exists to conduct a 
complete program selection, review and evaluation program 
(Ref. 9) . 
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5 . Operation — Correlation of Elements and Strategies 



In development of the strategies, the program element 
titles, which had been originally listed by the CND, were 
expanded to the current total of 37 (Ref. 9) . Although this 
increase facilitated submission by the cognizant internal 
agency, its effect is to fracture the program and make more 
difficult their aggregation in an overall plan. Figure 5 
shows the relationship which now exists between program ele- 
ments (PEs) , technical strategy areas and the Science and 
Technology Objectives (STOs) . It is noted that there is a 
high degree of correlation between PEs and strategies except 
in sea control and power projection where the strike warfare, 
countermeasures and two other program elements are split be- 
tween sea control and projection. 

The Chief of Naval Development establishes and promul- 
gates the structural framework for planning, programming and 
budgeting the EDP (Appendix B) . This structure provides the 
link with DoD Program elements, defines the scope of work 
within specific areas, and identifies the organization within 
the Navy responsible for the planning, programming and 
budgeting. Appendix C is included to provide clarification of 
the fundation of the Exploratory Development program by delin- 
eating its new structural framework. 

The system described above is new this current fiscal 
year. It is quite naturally going to involve a great deal of 
"negotiating” between the staffs of NAVIiAT , laboratories and 
SYSCOMs, along with DNT, CND, and ASN (R&D) . It is essential 
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Figure 5. 

Relationship Between Program Elements, 
Technical Strategies/Substrategies, and STOs 
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that a firm understanding of the technical strategies, func- 
tional areas and program elements be had to augment an under- 
standing of the EDP objectives and transition steps. Only 
through a thorough knowledge of the inner-workings of the new 
system will any meaningful review, evaluation and assessment 
methodology be realized. As indicated, no established pro- 
gram or technique currently exists, but there certainly is 
potential for developing a viable methodology, particularly 
with the well structured and goal-oriented EDP which exists 
today . 
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IV. PROGRAM SELECTION AND EVALUATION IN NAVY 



EXPLORATORY DEVELOPMENT 

With a background presented of the Navy Exploratory 
Development program, its functions, its responsibilities and 
its policies for implementation, it is beneficial to investi- 
gate now the manner in which programs have traditionally been 
reviewed, selected and evaluated. It would serve the reader 
well to keep in mind the new EDP characteristics, particularly 
while going through this review. This will provide a deeper 
comprehension of why the methodologies to be discussed were 
considered viable, as well as why they eventually were evi- 
denced as not being able to adequately predict, as a manage- 
ment tool, as originally intended. 

A . BACKGROUND 

Technology base programs constitute a large number of 
projects in the systems acquisition process, falling some- 
where between pure research on one end of the R&D spectriim 
and full scale development on the other. In the early years 
of the Navy R&D sector, these technology programs operated 
with little or no outside interference to the performing 
organization. Ideas for solving Fleet related problems and 
exploiting technological opportunities were converted into 
exploratory development projects with relative ease (Ref. 7) . 

In these early stages of R&D project plans, task assign- 
ments and monitoring procedures were generally flexible and 
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tailored to the needs of the individual managers at all 
levels. This process was in keeping with the "cradle to 
grave" concept. 

More recently, a perceived need to provide guidance for 
technology programs was reflected in dociiments ranging from 
top level Joint Chiefs of Staff strategic planning documents 
to OPNAV's General Operational Requirements and CND Explora- 
tory Development Goals. But this guidance soon tended to 
be either too general, providing little guidance at all, or 
so specific that it inhibited exploration of relevant 
alternatives . 

Because the problems associated with this guidance and 
direction quite naturally are felt at all levels within the 
organizational structure, the problem of effective program 
selection and evaluation has thus remained a significant 
issue, particularly in the area of exploratory development. 

According to a recent House Armed Services Committee 
report, concern was expressed that "... just spending money 
on research and development does not guarantee a sound R&D 
program or good military hardware" (Ref. 13) . The committee 
further indicated that the Department of Defense could 
improve its efforts and its management through better tailored 
programs which would; 

• Select out weapons not cost effective 

• Avoid unnecessary duplication of effort 

• Place greater emphasis on product improvement 

• Provide more meaningful management review processes. 
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It is not surprising that a tremendous amount of attention 
is currently being given to the valuation of DoD programs 
in Research and Development in general and Exploratory Devel- 
opment in particular. 

Adding now to the complexity of this situation is the fact 
that throughout the past fifteen years, while the total budget 
for RDT&E,N appropriations has been steadily increasing, the 
relative percentages allocated to the technology base pro- 
grams (6.2 category) have been steadily decreasing. This 
situation is depicted in the bar chart representation of 
apportionments, shown as Table II. 

And so, just within the past decade, many scientific, and 
some not so scientific, approaches have been tried for poten- 
tial application to technology base program evaluation and 
determination of return-on- investment figures. Most of these 
scientific techniques have been tried more at the lower 
echelons of the Navy R&D organization, although few have seen 
any real application or effectiveness. The approach currently 
in vogue at the NAVMAT 03 level is built around the concept 
of "technology strategies." There appears to be merit in 
pursuing this technique, at least for a selected application. 

B. HISTORICAL APPROACHES 

One of the early comprehensive attempts to sort, categorize 
and evaluate R&D selection and program evaluation techniques 
was that done by Baker and Pound back in the mid-sixties (Ref. 
14) . This article points out that most were oriented toward 
the operations research-management scientist, but it does 
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Table II 



Research and Development Categories as 
Percent of Total RDT&E;M Appropriation FY62-FY73 




Note: These amounts are derived from the past year columns of the various Congressional 
Budget Submissions and are not precisely consistent with New Obiigational Authorizations. 
The difference results from reprogramming between program years, permitted in “no-year" 
appropriations. The data for each fiscal year's program is the latest category distribution 
developed for budget presentation and best reflects actual relative dollar size of the various 
categories for each fiscal year. FY 1962 was the fir^year in which this category classifica- 
tion scheme was systematically applied. 



Source: Thomas Leckey, Assistant Compiroller for Budget and Reports ONR, ir> letter of 
24 December 1974. 
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provide a sound base for presenting the early methods thought 
to have potential. It is interesting to note that a review 
of the majority of the docviments reviewed in this account 
covering this early era indicated that very few of the tech- 
niques presented actually ever saw any relevant usage. 

Keeping in consonance with the early organization and 
management of the exploratory development sector of R&D, pro- 
ject selection thus remained a relatively unstructured prob- 
lem. For the most part, the early methods just did not 
provide an adequate description of the R&D process. At this 
point, it might appear feasible to question not only the pre- 
dictive models but the process which the models were attemp- 
ting to simulate. While it was recognized that program 
selection was a sequential decision making process, practi- 
cality would certainly indicate that detailed criticism of 
these early models would be unwarranted. 

One important observation made, according to Baker, was 
that "...there has not been a satisfactory definition of the 
objectives against which a research program should be evalu- 
ated." The significance of this statement, while having 
clarity at this point, will take on new interpretations later 
in this section. Another key issue raised in these early 
studies, although it received little more attention than having 
been mentioned, was the idea that the "strategic need" some- 
how ought to fit into the organizational and market structure. 

The methods applied in the early stages of investigation 
consisted almost entirely of three major methods: decision 
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theory, economic analysis and operations research. Table III 
is included in an attempt to sviramarize the significant features 
of these methodologies. 



Table III 



Features of Common R&D 


Planning 


Models 




Feature 




Method 




Characteristic consideration 


[decision 

theory 


Econonic 

analysis 


Operations 

research 



1. 


Sequential decisions 


No 


No 


Yes 


2. 


Profit maximization 


Pa 2 ±ially 


Yes 


Yes 


3. 


Uncertainty (probability 
of success) 


Yes 


Yes 


Yes 


4. 


Relationships between criteria 


No 


Partially 


Yes 


5. 


Relationships between projects 


No 


No 


Partially 


6. 


Optimal allocation to project 


No 


Partially 


Yes 


7. 


Budget constraint 


Yes 


Yes 


Yes 


8. 


Skill & facility constraints 


No 


No 


No 


9. 


"Strategic need" for project 


Partially 


No 


No 


10. 


Competitor efforts 


No 


No 


No 



Area of applicability 



1. 


Basic research 


Yes 


Maybe 


No 


*2. 


Applied research 


Maybe 


Yes 


Maybe 


3. 


Development 


Little 


Moderate 


Definitely 



*Technology base programs. 

A prime observation of these methods and practices is that 
there exists a strong and definite need for testing to demon- 
strate feasibility of the predictive models. Without adequate 
demonstration, even the "best" descriptive models prove sus- 
pect under the close scrutiny of management. 
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C . MODERN APPROACHES 



Once the idea that the management process of project 
selection and evaluation was realized to have potential as 
being predictable from a management scientist approach, a 
profuse amount of literature emerged, quite naturally. A 
great many new ideas were postulated, some unique in their 
own right, although most were merely extensions of what had 
already been presented in the early years. 

An excellent study conducted by IDA provides an informa- 
tive synopsis of the modern techniques being employed (Ref. 
15). Although prepared in early 1972, the methods discussed 
are still being utilized, if only under slight improvement. 

Because the modern techniques have tended to align them- 
selves in the more strict, quantitative approach to program 
selection and evaluation, most of the discussion at this 
point will tend to follow along with that momentum. With the 
ensuing budget constraints, as previously mentioned, the 
patter associated with the modern approaches will necessarily 
allude to resource allocation as its primary objective. 
Realize, however, that the real issue is still the process of 
determining which programs are to be selected for support and 
how, then, can they be evaluated once initiated. 

Some problems are inherently embedded in the modern 
methodologies, owing to their strong alliance to the quanti- 
tative approach of evaluation. A strong underlying problem 
is that even with modern conveniences and sophisticated ana- 
lytical software and hardware, the utilization factor still 
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remains low (Ref 16) . According to Albala in his work on 
stage approaches to R&D projects; 

This condition can be partly ascribed to the fact 
that (a) most of the proposed methods involve models 
requiring much quantitative input, not readily avail- 
able within the company, and (b) in many instances 
the methods are based on techniques much too elaborate 
for the R&D manager ' s routine use . 

In subtle opposition to this belief is the posture taken 
by Trozzo of IDA (Ref. 15.) Accordingly, Trozzo feels that: 

Development of more quantitative methods for allo- 
cating resources within Exploratory Development should 
continue, however, oecause their rigorous structure 
should provide (1) a useful frame of reference for 
organizing information and regular liaison among system 
specialists, technologists, and research managers, and 
(2) a systematic procedure for searching out a develop- 
ment program that will best fulfill Defense goals. 

Albeit, included as Table IV a siommary description of the 
methods used for allocating resources, taken directly from 
the study by IDA. Its inclusion is useful in showing the 
definite trend to quantitative rationale and also to demon- 
strate that each organization utilizes a methodology unique to 
its particular operation, based to a large extent on its own 
needs and variables. 

Thus the techniques that have grown to occupy currently 
acceptable standards have adhered to the general conclusions 
of the earlier studies. To wit, there is a general recognition 
that each particular stage of R&D requires a set of inputs and 
constraints unique to its decision making process (Ref. 16). 
Another important observation of modern approaches is their 
recognition that models need be commensurate with the quality 
and quantity of data available. Finally, project success, to 
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Table IV 

Summary Description of Methods for 
Allocating Development Resources. 







Primary Objective 


Control Variables 




Value Measures 


Opera tional 
Requirements 


Method 


Devise an exploratory 
development program 
that: 


Management's levers 


Usefulness or con- 
tribution of program 
results 


Major jobs of the 
Services 


1. Industrial 
Ana log 


Resembles the pattern 
of technological de- 
velopment in U.S. in- 
dustry 


Funds spent in major 
technical areas 


None 


Industry groupings — 
weapon categories 


2. TORQUE 


Develops technologies 
having maximum "mili- 
tary value" 


Funds to be spent in 
next year on project 
tasks 


Subjective scaling of 
relative "military 
value" 


Operational capa- 
bility objectives 
(OCO's) assigned 
values 


3. Naval Ordnance 
Laboratory 


Develops technologies 
having maximum 
"value" for future 
Navy needs 


Funds to be spent on 
each technology in 
next year 


Subjective scaling of 
relative value for 
each factor 


Warfare categories — 
ta rge ts — mi li ta r y 
functions assigned 
values 


4. A.F. Flight 
Dynamics 
Laboratory 


Develops technologies 
having maximum "mili- 
tary value" for the 
Air Force 


Funds and manpower to 
be devoted to each 
exploratory develop- 
ment task 


Subjective scaling of 
relative effective- 
ness, timing, etc. 


Operations within 
different types of 
wars — assigned values 


5. Cornell 

Aeronautical 

Laboratory 


Consists of tasks 
with maximum "ex- 
pected value" to the 
Army 


Tasks that will be 
performed 


"Essentiality" of 
weapons developed 


No general opera- 
tional requirements. 
Background to qual- 
itative materiel 
development objec- 
tive (QMDO) 


6, Hercules 
Corporation 


Contributes most to 
future company profits 


Funds to be spent on 
individual develop- 
ment tasks 


Profits generated by 
development work 


Market characteris- 
tics j 

! 

1 

1 

_ . 1 


7. Army Missile 
Plan 


Supports preferred 
weapon concepts pro- 
jected for future 
deployment 


Weapons and funding 
of requisite tasks 


Discrete priorities 
among weapons and 
technical work 


Operational require- 
ments in documents by 
CDC 


8. A.F. Directorate 
of Laboratories 
Plan 


Provides the technol- 
ogy required for prob- 
able future systems 


Technical effort in 
each area 


None 


Desired capabilities 
defined by OSD and 
the Air Staff 


9. Army Research 
Plan 


Responds to long 
range concepts and 
objectives of the 
Army of the future 


Funding emphasis for 
elements of program 


Priorities 


Future operational 
needs in 56 opera- 
tional capability o^b- 
jectives 

( 


10. Another Service 
Method 


Supplies technical 
advances for ful- 
filling overall 
Service mission 


Funding of individual 
development projects 


Subjective scaling of 
merit of requirements 
and contribution of 
technical objectives 


29 operational re- 
qui rements — meri t 
weights 
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Table IV (Cont'd) 





Structural Features and Factors Considered 




Factors and Their Relationships 




Weapon Systems 


Technologies 


Development Tasks 


Costs 


Method 


Instruments for per- 
forming jobs 


Composition of 
systems 


VJork to achieve de- 
sired composition 


Resources consumed in 
work 


Industrial 

Analog 


Individual weapons — 
industry products 
(DoD expenditures — 
industry sales) 


Weapon-product life- 
times and technical 
complexity 


None 


BoDl expenditures in 
6.1, 6.2 and total 
RDT6E on technical 
fie Id- -company funded ’ 
R&D in industry 


. TORQUE 


Alternative weapons 
for each OCO 


Technological re- 
quirements of each 
v/eapon- -criticality 
to system rated 


Technical objectives 
sequenced in each 
technical require- 
ment 


Yearly and total 
funding for each 
project task 


. Naval Ordnance 
Laboratory 


Contained in explor- 
atory development 
goals (EDG) military 
worth 


Technology pacing 
parameter linked to 
EDG by relevance 
measure 


Implicit — different 
levels of pacing 
parameters 


Aggregate expendi- 
tures on technology 
related to index of 
advance for parameter 


. A.F. Flight 
Dynamics 
Laboratory 


Future A.F. flight 
vehicles from tech- 
nological war plan — 
weighted for each 
operational re- 
quirement 


Technologies linked 
to systems by ap- 
plicability factor; 
separate technology 
goals 


Confidence level 
gauge of achievement; 
contribution value 
links task to systems 
and technological 
goals 


1 

Funding for different ! 
levels of effort j 

! 


. Cornell 

Aeronautical 

Laboratory 


QMDO description of 
hardware. Weighting 
proposed 


Combinations of 
technologies de- 
scribe possible 
technical approaches 
to subsystems 


Ordered tasks nnakeup 
technical approach 
resource pattern 


Direct funds and 
other resources 
needed 


. Hercules 
Corporation 


Projected systems 
using technology; 
contract research — 
sales, margins 


Substitute and com- 
plementary tech- 
nologies in each 
system 


Technical goals of 
specific organised 
units of effort 


Funding for each pos- 
sible level of effort 


. Army Missile 
Plan 


Systems to fulfill 
operational require- 
ments — priorities 

1 

1 


Technical composition 
of system and subsys- 
tem concepts; general 
functional capabili- 
ties 


Labs plan tasks for 
technical goals; 
tasks rated by con- 
tribution 


Manpower, funding for 
planned tasks 


1. A.F, Directorate 
of Laboratories 
Plan 


System concept pos- 
sibilities — most 
probable systems — 
subsystems 


Technical planning 
obj ecti ves — tachnica 1 
gaps for systems; 
technical goals 


Program elements-- 
projects — tasks — 
technical ef fores 


Summary of resources 
for technical ef- 
forts — program ele- 
ments 


K Army Research 
Plan 


Not explicit--con- 
tained in long range 
technological fore- 
cast 

i 


Potential relevance 
of technologies in 
30 6.2 elements to 
each OCO 


237 projects in 6.2 
elements; extra Army 
effort 


Funding trends and 
planned support of 
elements — adequacy of 
support 


). Another Service 
Method 




[one 


Technical objectives 
related to opera- 
tional requirements 


750 tasks — weights 
reflect contribution 
of each to each oper- 
ational requirement 


Estimated funding re- 
quired to complete 
project 
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Table IV (Cont’d) 





; 










Constraints 


Decision Algorithim 






Timing 


Risk 


Method 


Time -phasing of re- 
sources and results 


Probabi3dty of actual 
outcomes differing 
from expected 


Limits on possible 
program options 


Choosing pattern of 
development work to 
be done 


1. Industrial 
Ana log 


None 


None 


None 


Not specific — DoD 
funding proportional 
to industry in some 
way 


2. TORQUE 


Weapon ^CC-- tech- 
nical objectives — 
funding (cimeliness 
•function) 


None 


Total budget 


ZiValue/lCost in de- 
scending order among 
project tasks 


3. Naval Ordnance 
La bora tory 


Not explicit — prog- 
ress possible in 
nexj budget year 

1 


None 


Total budget 


APayoff/ACost equated 
over all technologies 


4. A.F. FUght 
Dynandcs 
Laboratory 


Timi 
objf= 
by - 
tior 


ng of technical 
ctrlve, weighted 
imeldness func- 

i 


Probability that task 
output will fit sys- 
tem 


Total funds, in- 
house and contract 
engineers 


Linear program 


5. Cornell 

Aeronautical 

Laboratory 


Timing of Ql-IDO's 
tasks 


Probabi3dty of suc- 
cess for each task, 
techjiical approach, 
subsystem 


QMDO requirements, 
funds , other re- 
sources 


Decrease budget re- 
quired by eliminating 
low A"Exp. Value’V 
ACost among technical 
approaches 


6. Hercules 
Corporation 


Timeliness of market- 
ing and time required 
for technical goals 


Probability of pre- 
dicted sales, mar- 
gins, research needs, 
technical timing, and 
success 


Total budget 


Search possible 
shifts of funds among 
projects for mix 
giving maximum pro- 
fits 


7. Army Missile 
Plan 


Timing of funds, 
tasks, target dates 
of systems 


None 


None 


Funds distributed 
down list of tasks 
ordered by priorities 


3. A.F. Directorate 
of Laboratories 
Plan 


Desired capabilities 
set 'out in five year 
increments 


None 


None 


None 


9. Army Research 
Plan 


Operational require- 
ments out to 20 
years; development 
program in next 1-5 
years 


Implicit — funding 
that gives reasonable 
probability of reach- 
ing technical goal on 
time 


In-house capability, 
external efforts, 
directed work 


Qualitative analysis 
of shifts in emphasis 
warranted from prior- 
ities, relevance, 
support 


0. Another Service 
Method 


Projects set up to 
support operational 
requirements of 1982 


Probability of suc- 
cess of each task to 
reach technical ob- 
jective with funds, 
and in time 


Sponsored projects, 
in-house capability 


Rank projects by de- 
scending exp. value/ 
cost 



56 




4i' 




a large degree, is directly correlated with the magnitude of 
the resources committed, which certainly should not come as 
any surprise. 

D. IN RETROSPECT 

And so some general conclusions and statements can be 
gleaned from a review of these predictive models. Firstly, 
the worth-assessment process has not really been adequately 
structured by the models for the specific goals and purposes 
of the military R&D system, particularly the EDP. To a 
large extent, this has been due to a generation of philoso- 
phies which restrict the options for dealing with future 
world situations by proposing concepts that are extensions of 
the current force structure and generally use the current 
technological approach to accomplish mission tasks . Conse- 
quently, and as previously indicated, R&D activities tend to 
focus on meeting system requirements rather than on gener- 
ating innovative ways to accomplish mission tasks. All too 
often the R&D activities in the EDP which are evaluated are 
not independent, as is necessary for worth-assessment proce- 
dures, but are often interrelated activities. Consequently, 
the projects selected usually do not provide all the technol- 
ogies necessary to develop and deploy a desired system. 

Secondly, the models exercised did not adequately provide 
procedures to assure the inputs necessary to meet the 
assumptions and requirements of weighting criteria. That is 
to say, missions, goals, and operational requirements were not 
(are not) being defined in terms that are worth-independent 
(Ref. 17) . 



57 




4 " 




Additionally, different worth criteria and underlying 
assessment processes are required to assess different activi- 
ties. Each model must therefore be specifically tailored to 
a particular activity. Then too, the cost effectiveness of 
actually using the models is very difficult to evaluate. The 
true effectiveness should generally be subjectively evaluated 
in terms of how useful an R&D manager finds the results. 

The future of R&D planning models similar to those identi- 
fied thus far is not particularly promising in light of the 
observed methodological problems and managerial resistance to 
their use. The models just cannot, or have not been able to 
capture all the relevant and subtle aspects of the R&D and EDP 
planning process. The methodological problems preclude the 
use of such models even as decision information systems to aid 
managers, since their output is highly questionable. 

But organizations performing Exploratory Development within 
R&D must still select and evaluate among many possible activi- 
ties, however. These decisions can be improved by a better 
understanding of both the process of innovation and the Explor- 
atory Development planning process. It is important that 
organizations have systematic procedures and logical organi- 
zational structures to assure that the major planning tasks are 
effectively accomplished. Figure 6 has been suggested as a 
representative outline of the necessary elements of a contempo- 
rary planning process (Ref. 17) . To develop such procedures 
and structures, the organization must clearly define what it 
hopes to accomplish by performing the exploratory development 
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OBJECTIVES 




Conceptual outline of a long range research 
development planning process. 
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within R&D. No technique , model or methodology is likely 
to improve the effectiveness of a poorly or inadequately 
implemented planning process. 
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V. DETERMINING ROI IN THE EXPLORATORY DEVELOPMENT PROGRAM 



It is difficult to argue with the fundamental proposition 
that any large enterprise ought to seek evidence of the effec- 
tiveness of existing programs and proposed program alterna- 
tives as a basic guide to decision making. Yet few sound 
evaluation schemes and even fewer rigorous field experiments 
have actually been undertaken. Serious questions exist con- 
cerning the capacity of the government both to develop and 
use evaluative results. Much of the concern deals with the 
short supply of competent policy-oriented researchers both 
within and outside of the government, the inadequacy of avail- 
able methods and concepts for carrying out evaluative studies, 
and the shortage of organizations with the capacity to under- 
take large-scale evaluative activities. 

An evaluation method based on the tools of social research 
has been applied successfully to social action programs to 
determine program effectiveness. The basic principles of this 
methodology, while developed originally for social and educa- 
tion programs, actually are oriented toward any action program 
which can attest to a sound platform of goals and objectives. 
Thus, there appears to be a potentially viable technique 
which would be applicable to programs exhibiting outputs which 
present themselves in a somewhat unquantif iable manner. 

This section presents an overview of the methodology, 
called Evaluation Research (ER) , and attempts to show how it 
might be applied to programs within the Exploratory Development 
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Program. While a complete treatise and development of an 
evaluation study on an exploratory program is beyond the 
scope of this paper, it is felt that the ER technique could 
be applied in particular applications to yield realistic 
measures of effectiveness of technology programs. A presen- 
tation of the background for the basic elements of such a 
study follows . 

A. EVALUATION RESEARCH FOR THE EDP 

Evaluation is a rather elastic word but common usage 
would generally imply the notion of judging merit. Evalua- 
tion has frequently had explicitly political overtones. It 
is designed to yield conclusions about the worth of programs 
and, in so doing, is intended to affect the allocation of 
resources . 

A form of evaluation, called Evaluation Research, has 
been devised and used in the past on social programs, although 
in its strictest sense is concerned with finding out how well 
action programs work. It represents the application of social 
science research methods to discover information of importance 
to program practice and public policy. The tools of research 
are applied to make the judging process more accurate and 
objective. The evaluation thus establishes clear and specific 
criteria for success. 

The research process is particularly important when (1) 
the outcomes to be evaluated are complex, hard to observe, 
made up of many elements reacting in diverse ways; (2) the 
decisions that will follow are important and expensive; and 
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(3) evidence is needed to convince other people about the 
validity of the conclusions (Ref. 18) . Clearly then, this 
scheme appears to have commonality with the R&D process in 
general and the Exploratory Development Program in particular. 

The rationale of Evaluation Research is that it provides 
evidence on which to base decisions about maintaining, 
insituationalizing and expanding successful programs and 
modifying or abandoning unsuccessful ones. For example, the 
methodology of determining program selection at a laboratory 
Block Program Office utilizes a decision matrix, as shown in 
Table V, provides much of the necessary basic information 
for initiating an evaluation research technique. The matrix 
is utilized as a tool for determining which programs should 
be supported and continued, as they relate to mission needs. 
Used in this context, evaluation research is increasingly 
important as a source of knowledge and direction. It can be 
used to tell which programs work and which do not, and points 
the way to better formulation of policy and program. It can 
be even more helpful as it proceeds to identify the effects 
of specific strategies and components within programs and 
separates out those that contribute to favorable outcomes 
from those that are ineffective or counter productive (Ref. 

18) . 

If evaluation and cost-benefit analysis are to have any 
effect on decisions , there has to be some way to move from 
the data to the decision — some system for transmitting 
information and feeding it into the decision process. The 
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Table V. Block Program Decision Matrix 



PRIORITIES 


SYSTEMS CONCEPT 


1 


SR A AM 


3 


SELF-DEFENSE 


6 


AIR-TO-AIR ATAAM 

MISSILES A[AAM 


2 


AIAAM 


3 


ALRAAM 


1 


ANTI-SHIP 


1 


°nRE^ air-to-ground 


3 


DEFENSE SUPPRESSION 


1 


ANTI-SHIP 


1 


"^fIre*^ air-to-ground 


3 


DEFENSE SUPPRESSION 


1 


ANTI-SHIP 


1 


M?ssHI air-to-ground 


3 


ANTI-SHIP MINE 


1 


TARGET ACQUISITION/ SHIP 


1 


CLASSI FICATION SYSTEM LAND 


1 


ADVANCED AIRBORNE ARMAMENT SYSTEM 


2 


NIGHT/ADVERSE WEATHER 


2 


FIRE AIR-SURFACE LIGHTWEIGHT 


5 


CONTROL AIR-TO-AIR GUN 


5 


HELICOPTER 


6 


AIRBORNE AIR-SURFACE LIGHTWEIGHT 


5 


GUNS ADVANCED 


5 


CLUSTER WEAPONS ADVANCED 


5 


LIGHTWEIGHT 


4 


SnrRAP«-0N G&C KITS (KMU) 


4 


ADVANCED BOMB FAMILY 


4 


CHEMICAL WEAPONS 
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Planning, Programming, Budgeting System (PPBS) recently in 
use in DoD is such a system, but it starts with the definition 
of objectives rather than the delineation of goals. Providing 
a systematic approach to governmental planning, the distinctive 
characteristics of PPBS are: 

1. It identifies and defines fundamental objectives and 
considers all activities in terms of those objectives. 

2. It explicitly and systematically identifies alterna- 
tive ways of carrying out the objectives. 

3. It estimates the total cost implications of each 
alternative, looking not only at the current year but also 
future years . 

4. It estimates the expected benefits of each alternative. 

5. It presents the resulting cost and benefit comparisons 
for each alternative, along with identification of major 
assiamptions and uncertainties. 

PPBS is more concerned with future options than with past 
experience, especially when applied in the technology program 
area. But in estimating expected benefits, it has to derive 
its figures from somewhere, and this is where evaluation 
should come in . 

B. PURPOSE OF EVALUATION RESEARCH 

Evaluation research is viewed by many as a way to increase 
the rationality of decision making and policy making. With 
objective information on the outcomes of programs, wise 
decisions can be made on budget allocations and program 
planning. Programs that yield good results will be expanded; 
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those that make poor showings will be abandoned or drasti- 
cally modified. 

An evaluation study does not generally come up with final 
and unequivocal findings about the worth of a program. Its 
results often show small, ambiguous changes, minor effects, 
outcomes influenced by the specific events of the place and 
the moment. It may require continued study over time and 
across projects to speak with confidence about success and 
failure . 

For decision makers, evaluation evidence of outcome is 
only one input out of many. They must consider a host of 
other factors, from public receptivity to participant reac- 
tion, costs, availability of staff and facilities, and 
possible alternatives. Evaluation will definitely not take 
politics out of the decision making process. 

The purpose of evaluation research is to measure the 
effects of a program against the goals it set out to accom- 
plish as a means of contributing to sv±>sequent decision 
making about the program and improving future programming 
(Ref. 19). . The measurement of the effects refers to util- 
izing a research methodology to investigate outcomes of the 
program rather than efficiency or adherence to standards. 

The comparison of effects with goals stresses the use of 
explicit criteria for judging how well the program is doing. 

Evaluation research is committed to the principle of 
utility. If it is not going to have any effect on the 
decision, then it is clearly an exercise in futility. Weiss 
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(Ref. 19) suggests that there are probably four kinds of 
circumstances in which evaluation is probably not worth doing: 

1. When there are no questions about the program. It 
goes on, and decisions about its future either do 
not come up or have already been made. 

2. When the program has no clear orientation. Program 
staff improvise activities from day to day, based 
on little thought and less principle, and the pro- 
gram shifts and changes, wanders around and seeks 
direction. 

3. When people who should know cannot agree on what the 
program is trying to achieve. If there are vast 
discrepancies in perceived goals, evaluation has no 
ground to stand on. 

4. When there is not enough money or staff sufficiently 
qualified to conduct the evaluation. Evaluation is 
a demanding business, requiring time, money, 
imagination, tenacity and skill. 

The all-purpose evaluation is a myth. But there are several 
types of uses for evaluation. Evaluation can investigate the 
extent of program success so that decisions such as these can 
be made: 



1. To continue or discontinue the program. 

2. To improve its practices or procedures. 

3. To add or drop specific program strategies and 
techniques . 

4. To institute similar program elsewhere. 

5. To allocate resources among competing programs. 

6. To accept or reject a program approach or theory . 



C. ELEMENTS OF AN EVALUATION RESEARCH PROCESS 

In abstract, the basic elements of the evaluation research 
process include (1) finding out the program’s goals, (2) trans- 
lating the goals into measurable indicators of goal achievement. 



67 




i 





(3) collecting data on the indicators of the participating 
elements (and for an equivalent control group) , and (4) 
comparing the data on participants (and control elements) 
with the goal criteria. 

But this process in theory becomes much more complex in 
practice. A host of unexpected problems invariably arise 
such as : 

1. Program goals are often hazy, ambiguous, hard to pin 
down. 

2. Programs not only move toward official goals, but they 
also accomplish other things, sometimes in addition and some- 
times instead. 

3. The program is a congeries of activities, people and 
structures, some irrevelant. 

4. The evaluation question as posed ignores the issue of 
why the program succeeds or fails. 

Ideally then Weiss (Ref. 18) suggests the folowing as 
the necessary core issues to comprise the effective evaluation 
research process; 

1. Formulate the program goals that the evaluation will 
use as criteria. 

2. Choose among multiple goals. 

3. Investigate unanticipated consequences. 

4. Measure outcomes. 

5. Specify what the program is. 

6. Measure program inputs and intervening processes. 

7. Collect and compare necessary data. 
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In its rudimentary form then, the evaluation process 
encompasses the interpretation and implementation, including 
the elements of policy, goals, theory, models, experiment 
(to check theory and provide criteria for success-failure) . 

The experimental model remains the ideal in evaluation 
methodology, with random assignment of subjects to an experi- 
mental group which is exposed to the program stimulus and to 
a control group which is not, thus providing the research 
rationale. 

It collects before and after measures over a full cycle 
of the program and uses specified goal criteria. Its results, 
then, disclose the extent to which a consistent program has 
reached its stated goals. However, it rarely distinguishes 
why the observed results occur, what processes intervene 
between input and outcome or what the particular implications 
are for improving the effectiveness of the program. 

The application to the EDP, however, might just provide 
the necessary means with which to conduct a satisfactory 
selection/evaluation process, ultimately yielding a rational 
method for determining program worth. The stage is set; what 
remains now is a progressive-oriented faction of the Explora- 
tory Development Programs , willing to commit the necessary 
resources of time, personnel and money to explore the poten- 
tial of this new technique. 
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VI. SUMMARY AND CONCLUSIONS 



The Navy Exploratory Development Program is currently 
undergoing major revision to align its management planning 
system with the philosophies of the new wave of thinking 
regarding program effectiveness; i.e., to meet the Navy's 
basic obligation to put government resources to their most 
effective use in mission accomplishment. 

Traditionally, technology has gone on to develop subsystems 
and systems concepts independently of specific mission needs. 
Further, rather than actively building a strong technology 
base through innovation, old technologies have been extended 
to improve performance of old kinds of products . New systems , 
while becoming more complex and expensive have actually become 
less technologically advanced. 

It is recognized that it is imperative that cost-benefits 
be maximized on technology base programs. This, though, can 
only be realized when the base programs are driven from oper- 
ational needs and mission requirements. 

A complete and sound understanding of the functions and 
underlying philosophies and policies of the technology pro- 
gram is essential to determine that program's worth, or 
return-on- investment . 

Sound foundations must be established from which goals and 
strategies can be derived to provide meaningful inputs for 
evaluation methodologies of contemporary exploitation, rather 
than basing output measures on dollar values. 
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In terms of the policy platform for the Exploratory Devel- 
opment Program, the underlying principles evidenced in the 
new rationale include: 

• Supportive of technological advances being of funda- 
mental importance to operational needs . 

• Quest for technical and managerial excellence at all 
levels . 

• Recognition of the necessity for both top-down direction 
and bottom-up inputs . 

• Accountability essential at all levels. 

• Necessity for strategies as tools to implement policies. 

• Recognition that sometimes technical advance must be 
injected into the main stream of the system forcibly. 

• Recognition of essential need for adequate information 
systems to allow interaction. 

With a strong push in Navy policy to promote accountability, 
a more decentralized organization is developing, as evidenced 
by the formation of the Block Program Offices. This allows 
a means for providing technical management in the laboratories 
and Systems Commands with the necessary authority to plan and 
manage technical efforts related to the pursuit of significant 
objectives . 

As a result of direction by the ASN (R&D) to establish and 
implement a program to improve the exploratory development sys- 
tem, to improve higher echelons' understanding of the system 
and to justify 6.2 expenditures, the Exploratory Development 
Program has been, and currently is being radically revamped. 
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Central to these modifications is the use of dynamic "technical 
strategies,” designed to focus exploratory development more 
directly and visibly on operational needs. The primary objec- 
tives of the new management system are to; 

• Provide top-down direction by the use of technical 
strategies, prioritized technical objectives and 
definition of major program thrusts designed to meet 
operational requirements . 

• Provide visibility by relating the program to Navy 
functions and operational needs. 

• Promote innovation and concentration on highest leverage 
opportunities. 

• Expedite transfer of projects to advanced development 
(6.3) and use of new technical capabilities developed 
in research (6.1). 

• Demonstrate program relevance and cost-effectiveness. 

It is essential that a firm understanding of the technical 
strategies , functional areas and program elements be had to 
augment an understanding of the EDP objectives and transition 
steps . Only through a thorough knowledge of the inner work- 
ings of the new system can any meaningful review, evaluation 
and assessment methodology be realized. While no established 
program exists currently , there certainly is potential for 
developing a viable methodology, particularly with the well 
structured and goal oriented EDP being implemented today. 

Worth-assessment processes to date have not been able to 
be adequately applied to the R&D structure in general, nor the 
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EDP in particular. This is due primarily to the traditional 
programs being system requirement rather than mission need 
(goal) oriented. Further, earlier models did not define in- 
puts in terms that were worth-dependent because different 
worth criteria and underlying assessment processes must be 
specifically tailored to particular activities. 

The future of R&D planning models is not particularly 
promising in light of observed methodological problems and 
managerial resistance to their use. The models just have not 
been able to capture all of the relevant and subtle aspects 
of the R&D and the EDP planning process . 

But organizations performing exploratory development must 
still select and evaluate programs some way. These decisions 
can be improved through a better understanding of both the 
process of innovation and the exploratory development planning 
process . 

New techniques and methodologies, as well as revisions of 
old ones, are constantly being identified, many receiving par- 
ticular area utilization, which will contribute to the process 
of rational decision making; i.e., program evaluation and 
assessment. One method felt to have high potential in the 
exploratory development area is that called Evaluation Research. 
Originally developed for application to social action programs, 
it is best utilized in situations where sound platforms of 
goals and objectives can be established. Thus, there appears 
to be a potentially viable technique which might be applied to 
programs in the EPD which yield outputs in a somewhat unquanti- 
fiable manner. 
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The stage has been set for the design of a new methodo- 
logical tool to assess the worth of programs in the EDP. 

A technique has been identified which has high potential. It 
is strongly recommended that further study be undertaken, 
using the background and management planning systems presented 
in this thesis, to ultimately demonstrate the ability of eval- 
uation research to effectively determine a measure of worth 
of programs in the Exploratory Development Program. 

The evidence suggests that such a tool can be realized. 
What is required now is to uncover a progressively oriented 
faction of the EDP willing to commit the necessary resources 
to more fully explore and demonstrate the potential of this 
research technique. 
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APPENDIX A 



EXPLORATORY DEVELOPMENT PROGRAM RESPONSIBILITIES 

The following delineates Exploratory Development Program 
management responsibilities. It discusses basic organization 
design principles, the responsiblities of Program organiza- 
tions and officials , and the various roles played by non- 
NAVMAT organizations and officials in the 6.2 Program. 

Organization Design Concepts . The Exploratory Development 
Program management system is designed to strike the best com- 
promise between the need to administer public resources in 
accordance with applicable laws, regulations, and policies, 
and the need to provide technical management with the freedom 
required to hold them accountable for results. The responsi- 
bilities described for the Exploratory Development Program are 
based on the following design principles: 

Fixed accountability . Establishing the conditions neces- 
ary to hold individual officials (as distinguished from diffuse 
bureaucracies) accountable is a central consideration. Pre- 
requisites to accountability include: 

a. Assigning responsibility for results to an individual. 

b. Provide that individual with the necessary authority 
and resources to accomplish the required results. If a mana- 
ger is to be held accountable, he must have the opportunity 
to pursue objectives by the means which he judges most 
effective. 
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c. Providing the responsible individual with the oppor- 
tunity to perforin by limiting the demands on his time. This 
includes demands by superiors and their staffs, and by other 
external officials and organizations. 

Clear separation of responsibilities . Responsibilities 
for determing the objectives to be pursued, and the technical 
effort required to fulfill those objectives, have been clearly 
and deliberately separated. Although it is essential that 
spokesmen for these two interests conduct an in-depth dia- 
logue in the process of defining the Program, it is important 
to avoid conflict of interest. In the management system 
described herein, technical people are not in a position to 
dictate "requirements," and management people and sponsors 
are not in a position to control the selection of technical 
approach. 

Quality information for management . The management sys- 
tem has been designed to provide the ASN (R&D) , CND, and 
lower-level officials with all the information they require 
to implement their Program responsibilities. The interaction 
of organizations generates quantitative information required 
for technical management to put technical resources to their 
most productive use, and for higher-level management to 
reallocate resources as required to increase overall Program 
output. A system of institutionalized Program Reviews ensures 
the quality of information provided to top management. 
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Management by objectives and results. The management 



system relies on mutually established and agreed-upon objec- 
tives and goals rather than on management by specific direc- 
tives, detailed procedures, elaborate reporting systems, 
frequent reviews, and other close controls. Management is 
by "self-control” in pursuit of the most efficient accomplish- 
ments of stated objectives. Higher-level review primarily 
measures results achieved in relation to approved objectives. 

"People information system" for coupling . The process of 
defining EDP projects which collectively support Department 
of the Navy purposes requires extensive and intimate inter- 
action between competent spokesmen for both sponsor and per- 
former organizations. The interaction of Technical Strategists 
with spokesmen for Mission Needs and with technical managers 
in the laboratories is a vital element of the "people infor- 
mation system." This face-to-face, paper-free information 
system provides a mechanism for the flow of information on 
needs and operational environment to the technical community, 
and for the reverse flow of knowledge about technical oppor- 
tunity to the CNO/CMC staffs. 

Program Management Responsibilities 

Deputy Chief of Naval Material (Development) (DNM(D)/ 

Chief of Naval Development (CND) ) . As DCNM(D) , this officer 
reports directly to CNM. As CND, he is directly responsible 
to ASN (R&D) . DCNM(D)/CND has overall control of the Explor- 
atory Development Program and its funds. He carries out this 
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general responsibility primarily through the DNT. His func- 
tions include: 

a. Reviewing and approving the Exploratory Development 
Program Plan (EDPP) at each stage in its evolution (i.e., the 
Draft and Tentative EDPPs, as well as the EDPP- 

b. Appointing Technical Strategists. 

c. Approving Technical Strategies. 

d. Approving Program Plans and related budgets. 

e. Directing the allocation of funds. 

Director of Technology Programs /Director of Navy Technol - 
ogy DTP/DNT . As DNT, this official is directly responsible 
to the ASN (RScD) . As DTP, he is responsible to the DCNM(D)/ 

CND. Acting for the CND, he is responsible for the planning, 
execution, and evaluation of the Exploratory Development Pro- 
gram. His functions include the following; 

a. Designing a balanced and integrated overall Explora- 
tory Development Program which maximizes the contribution of 
the Technology Base to the mission capability of Department 
of the Navy operating forces. 

b. Developing and documenting the Exploratory Development 
Program Plan (EDPP) for review and approval by CND and ASN (R&D) . 

c. Nominating Technical Strategists and appointing 
Strategy Team members. 

d. Reviewing and coordinating Technical Strategies. 

e. Developing a Program-wide priority list for concept- 
related technical effort. 
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f. Negotiating Program Plans (and related budgets) with 
Systems Command Directors of Research and Technology (SYSCOM 
03s) and with Technical Directors of the laboratories (Lab 
TDs) , and recommending those plans to CND for approval. 

g. Appointing Program Review Team leaders, and assisting 
in the staffing of Review Teams. 

h. Scheduling and chartering External Program Review. 

i. Conducting Program Reviews at the SYSCOM 03/Lab TD 
level. 

SYSCOM Directors of Research and Technology (SYSCOM 03s ) . 
The SYSCOM 0 3s plan and execute that portion of the overall 
Exploratory Development Program related to System Concepts 
(SCs) and other products within the material responsibility 
areas of their SYSCOM. The general responsibilities of a 
SYSCOM 03 include: 

a. Designing a balanced and integrated overall SYSCOM 
Exploratory Development Program which supports Objectives in 
the approved Mission Area Technical Strategies. 

b. Documenting planned technical efforts in a Program 
Plan for negotiation with and approval by the DNT. 

c. Prioritizing concept-related objectives set forth in 
the Program Plan. 

d. Providing designated personnel to serve as Technical 
Strategists and as members of Strategy and Program Review 
Teams, as requested by DNT. 

e. Negotiating and approving proposed TAPs. 
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f. Reviewing proposed changes in TAPs approving desired 
changes within his authority, and sxibmitting desired changes 
beyond his authority to DNT for appropriate action. 

g. Monitoring execution of his SYSCOM's technical program, 
and taking actions as appropriate. 

h. Ensuring the Internal Review of his SYSCOM's program, 
and providing copies of Internal Review Reports to DNT. 

Laboratory Technical Directors (Lab TPs ) . The Lab TDs plan 
and execute that portion of the overall Exploratory Develop- 
ment Program related to the development and utilization of 
technology within the areas of Laboratory responsibilities. 

The general responsibilities of the Lab TD include the follow- 
ing functions; 

a. Designing a balanced and integrated overall Laboratory 
Exploratory Development Program which supports approved Tech- 
nology Area Strategies. 

b. Documenting planned technical efforts in a Program 
Plan for negotiation with and approval by the DNT. 

c. Prioritizing technology-related objectives set forth 
in the Program Plan. 

d. Providing designated personnel to serve as Technical 
Strategists and as members of Strategy and Program Review 
Teams, as requested by DNT. 

e. Negotiating and approving proposed TAPs. 

f. Monitoring execution of his Laboratory's Technical 
Program, and taking actions as appropriate. 
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g. Reviewing changes in proposed TAPs approving desired 
changes within his authority, and submitting desired changes 
beyond his authority to the DNT or SYSCOM 03 as appropriate. 

h. Ensuring the Internal Review of his Laboratory's pro- 
gram, and providing copies of Internal Review Reports to DNT. 

Responsibilities for preparation of Technical Strategies . 
Each Technical Strategy is developed and/or updated by a Tech- 
nical Strategist and a supporting Technical Strategy Team. 

Technical Strategists . A Technical Strategist is respon- 
sible for the development and updating of each Technical 
Strategy. Strategists report to DNT for all matters relating 
to their Technical Strategies. Technical Strategists may be 
recruited from any Department of the Navy organization; 
normally, however, they will be NAVMAT, Laboratory, or Systems 
Command officials "double hatted" to CND for this duty. The 
general responsibilities of Technical Strategists include: 

a. Identifying Strategy Team needs with the appropriate 
knowledge and organizational affiliations. 

b. Developing and/or updating the assigned Technical 
Strategy as required (with the aid of the Strategy Team) . 

c. Maintaining current knowledge of the Technical Strategy 
in general (and its MA or TA Objectives in particular) and of 
all relevant technical capabilities, current efforts, and 
planned efforts. 

d. Reviewing all TAPs which purport to support Objectives 
listed in the Strategy, and taking appropriate actions to im- 
prove the effectiveness of technical contributions to higher- 
level objectives. 
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e. Serving as person-to-person channels between CND/DNT, 
Strategy Teams, and the technical community for the flow of 
needs information related to their Technical Strategies. 

This function includes helping technical personnel establish 
direct contact with individuals most knowledgeable about issues 
of interest. 

Strategy Team . Strategy Teams are constituted so that 
they have within the Team the essential knowledge and expertise 
required for the development of the Technical Strategy — without 
reference to documents except for detailed information. Such 
representation, in addition to helping in the cost-effective 
preparation of an adequate Strategy document, is an essential 
element in the development and maintenance of the "people 
information system." Ideally, the Strategy Team includes 
members from organizations which both require information on 
the Strategy, and which are sources of information required in 
the development of the Strategy. Each Strategy Team should 
include individuals who served on the Team in prior years; 
this ensures the preservation of a paper-free institutional 
memory. MA Strategy Teams include members from all TA Strategy 
Teams for technologies critical to the attainment of the MA 
Objectives. Participation of these people provides a direct 
and paper-free injection of information on technological capa- 
bilities and potential capabilities into the development and 
updating of the Strategy, and a reverse flow of Mission Need 
information into the program planning and execution activities 
of their parent Systems Commands and Laboratories. TA Strategy 
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Teams include spokesmen for SCs which use subject technology, 
members from organizations developing svibject technology, and 
representatives of the research community involved in develop- 
ing the base of knowledge underlying the subject technology. 

Source of Strategy Team members . Strategy Team members 
are recruited from both Government and the private sector. 
Department of the Navy technical organizations are directed 
to provide members to Technical Strategists on a negotiated 
basis. Assistance in recruiting non-Department of the Navy 
personnel will be provided by DNT (Attention MAT 03T3) . 
Strategy Team members from Department of the Navy organiza- 
tions are to be provided travel and analytical support by 
their parent organizations as required. 

Program Review Teams . There are two kinds of Program 
Review: Internal Review and External Review (see Chap. IV) . 

Internal Review Teams, under the leadership of the cognizant 
Laboratory Technical Director or SYSCOM 03, conduct self- 
reviews of all detailed technical programs. External Review 
Teams, under the leadership of a designated member of DNT ' s 
staff, conduct External Program Reviews at the Laboratory/ 
SYSCOM level. 

Staffing the Interval Reviews . SYSCOM 03s and Laboratory 
Technical Directors are responsible for determining the size 
and make-up of each Laboratory or SYSCOM Internal Review 
Team, and for assigning individuals from within their respec- 
tive organizations to serve as Team members. TD/SYSCOM 03s 
also have the option of recruiting Team members from outside 
their organizations. 
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staffing External Reviews. Two primary considerations 



determine the staffing of External Review Teams; (1) Team 
members should be qualified to verify the validity of Internal 
Review information without reference to additional documenta- 
tion, and (2) the External Review Process should establish 
a "people information system" which contributes to the effec- 
tive planning and execution of the overall Program. The DNT- 
chartered External Program Review Team is a broad-based ad hoc 
group. DNT appoints an External Review Team leader from his 
staff; he also appoints other Team members from among inter- 
ested and qualified Navy and non-Navy organizations. External 
Review Team members include; (1) current and/or former members 
of relevant Strategy Teams, (2) DNT staff members, (3) members 
of previous DNT Program Review Teams (to ensure the preserva- 
tion of a paper-free institutional memory; these individuals 
can note what has been done about deficiencies discovered in 
previous External Program Reviews) , (4) representatives of 

organizations which supply inputs to the programs under review 
(e.g., ONR Scientific Officers for disciplines critical to the 
technologies involved) , (5) representatives of organizations 

which use the outputs of the programs under review, (6) tech- 
nology managers who conduct similar programs, and (7) experts 
from the wider technical community, including industry. Extern- 
nal Program Review Team members are important links in the 
paper-free "people information system." 

DNT Staff . The DNT staff provides the DNT with supporting 
services, information, and counsel required for effective 
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performance of DNT's Exploratory Development Program manage- 
ment functions. Individual staff members are assigned cogni- 
zance over specific matters — such as individual Technical 
Strategies, Laboratories and SYSCOMs and their related Pro- 
gram Plans, and Program Elements. Specific duties of these 
cognizant staff members are discussed below. 

Technical Strategies . Duties of the DNT staff members 
assigned cognizance over a Technical Strategy include: 

a. Achieving and maintaining in-depth knowledge of the 
Technical Strategy, its rationale, and of all on-going or 
planned technical effort supporting Objectives in the approved 
Strategy . 

b. Serving as a member of the Strategy Team for that 
Strategy . 

c. Providing staff services for the recruitment and 
assignment of Strategy Team members, in compliance with staffing 
requirements developed by the Technical Strategist. 

d. Alerting the DNT to developments or trends related to 
the Strategy which require DNT’s attention. 

Organization and Program Plans . Duties of DNT staff mem- 
bers assigned cognizance over technical organizations and 
related Program Plans include: 

a. Achieving and maintaining in-depth knowledge of the 
planning and execution aspects of the organization's technical 
program — including all changes to the originally approved 
program. 
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b. Reviewing proposed Program Plans and providing DNT 
with appropriate information and counsel. 

c. Serving on the Program Review Team (as leader or 
member) . 

d. Determining requirements for Review Team personnel, 
and taking necessary actions to facilitate the recruitment 
and assignment of appropriate personnel. 

e. Alerting the DNT to developments or trends related 
to the organization (or its Technical program) which require 
DNT ' s attention. 

f. Providing oral and written information about the 
organization and its program to Government officials and pri- 
vate sector individuals with a need to know. 

Program Elements . Duties of DNT staff members assigned 
cognizance over a Program Element include: 

a. Achieving and maintaining in-depth knowledge of: 

(1) planning and execution aspects of work included within the 
Program Element, (.2) technical objectives and supporting 
rationales for the work performed, and (3) all actions and 
policies of higher level officials, including the Congress, 
related to the Program Element. 

b. Preparing Program Element Descriptions (PEDs) and any 
other documentation required in connection with the annual 
budget submission, apportionment process. Congressional 
hearings, etc. 
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c. Preparing PE-level TAP Aggregates using TAP informa- 
tion approved by SYSCOM 03s and Lab TDs (and by DC/S RD&S and 
CG MCDEC for Marine Corps programs) . 

d. Alerting DNT to developments or trends related to the 
Program Element which require DNT's attention. 

e. Providing information, both orally and in writing, 
concerning the Program Element to Government officials and 
private sector individuals with a need to know. 

Interfacing Responsibilities . Other organizations and 
individuals outside of the Naval Material Command play signifi- 
cant roles in the Exploratory Development Program. 

Congress . The Congress provides the funds for the 6.2 
Program and establishes funding controls at the Program Ele- 
ment level. Congress may, on occasion, control the funding of 
individual projects. Management responsiveness to Congress, 
credibility of information, and utility of results are essen- 
tial to continued stability of the Navy's Program. 

Director of Defense Research and Engineering (DDR&E ) . The 
DDR&E exerts influence at all levels of the Program. He can 
compare the management and technical content of the Exploratory 
Development Programs of all Services, and can direct all tech- 
nical programs in an area to be assigned to one Service. The 
DDR&E is the dominant DoD agency in the justification of work 
and the budgeting of funds . 

Assistant Secretary of the Navy (Research and Development 
CASN CR&D ) ) ♦ The ASN (R&D) is responsible for the RDT&E(N) 
appropriation, and for the planning and execution of the entire 
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Department of the Navy RDT&E Program (including Exploratory 
Development) . ASN (R&D) reviews and approves the various 
versions of the EDPP on an exception basis. He is also charged 
with justification of the Navy EDP to DDR&E and Congress. The 
small ASN (R&D) staff relies heavily on the CND and his staff 
for information and presentation of the program. Close work- 
ing relationships and mutual confidence between CND/DNT and 
the ASN (R&D) are essential. 

Chief of Naval Operations/Commandant of the Marine Corps 
( CNO/CMC ) . The CNO and CMC influence the 6.2 Program by estab- 
lishing operational needs and by directly sponsoring Advanced 
and Engineering Development. As the primary markets for the 
Program output, the CNO/CMC must be satisfied with its mili- 
tary relevance. The CND, assisted by the SYSCOMs, interprets 
the operational needs described by CNO/CMC in the Technical 
Strategy docxaments. The subsequent discussion, understanding, 
refinement, and definition of operational needs lead to devel- 
opment and acceptance of a useful product. CNO/CMC review 
all SCs when they are first submitted, and annually as long 
as they are included in the Program. CNO/CMC estimate the 
military worths of all SCs and assign them quantitative mili- 
tary worth ratings, "V" ratings, for use in planning Technical 
Strategies and supporting technical programs. 

Chief of Naval Research (CNR ) . In addition to his respon- 
sibilities for the Naval Research Program, the CNR is responsi- 
ble for sponsoring these exploratory development projects 



88 



which are necessary to ensure orderly transition from research 
to development. For the projects which he sponsors, CNR func- 
tions like a Systems Command. 

Chief of Naval Personnel (CNF) and Chief, Bureau of 
Medicine and Surgery (CHBUMED ) . CNP/CHBUMED are the primary 
sponsors for Exploratory Development relative to the human 
subsystem and to medical and dental systems. They are under- 
stood throughout this instruction to function like a Systems 
Command in these areas. 

Indus try /Academia . Industry and educational institutions 
interact principally at the SYSCOM and Laboratory levels. 

They provide Laboratories and SYSCOMs with assistance, on a 
contract basis, in developing both technologies and products. 
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APPENDIX B 



EXPLORATORY DEVELOPI4ENT PROGRAM DOCUMENTATION 

The documentation for planning and justifying the Explora- 
tory Development Program is described in this appendix. The 
basic document through which the Chief of Naval Development 
(CND) and the Director of Navy Technology (DNT) control the 
Program is the Exploratory Development Program Plan (EDPP) . 

The EDPP is supported by Technical Strategy documents, Labor- 
atory and System Command Program Plans, and Systems Concepts 
(SCs) . 

Exploratoiry Development Program Plan (EDPP ) . The EDPP 
is the authoritative statement of the content and rationale 
of the Navy's Exploratory Development Program. It defines 
the major thrusts of the overall Program, identifies major 
Program goals, and apportions resources among all elements 
of the Program. Further, it summarizes and provides an inte- 
grated overview of the Technical Strategies and the supporting 
Laboratoiry and Systems Command Program Plans. The EDPP is 
prepared by the Director of Navy Technology (DNT) on the basis 
of: (1) budget and program guidance from higher authority, 

(2) Technical Objectives developed by the Strategy Team, 

(3) Program Plans developed by Laboratories and SYSCOMs, and 
(.4) Internal and External Program Review Reports. The EDPP 
is continually updated to reflect changes in these inputs. 

Technical Strategies . Technical Strategies serve two 
major purposes: They effectively relate the Exploratory 



90 



Development Program to Department of the Navy Mission Needs, 
and they make that relationship visible and credible. 

Functions of Technical Strategies . Technical Strategies 
have both direct and indirect functions. 

Direct Functions . Technical Strategies are designed to; 

(1) take maximum advantage of new technical opportunities, 

(2) exploit deficiencies in opposition capabilities, and 

(3) provide prompt response to perceived requirements for 
superior naval capability. 

Indirect Functions . Strategies provide the vehicle for a 
two-way information flow on needs and technological capabili- 
ties (and possibilities) . They also serve as the means for 
demonstrating the general rationality — and specifically the 
mission-relevance in output-oriented terms — of the Exploratory 
Development Program. 

Kinds of Technical Strategies . There are two kinds of 
Technical Strategies; those which relate to the Mission 
Areas for which the Navy has responsibility and those which 
relate to the Technology Areas which are basic to the Navy's 
operational capabilities. 

Mission Area (MA) Technical Strategies . MA Technical 
Strategies directly link the Exploratory Development Program 
to the major missions of Department of the Navy operating 
forces. MA Strategies are developed for assigned Mission 
Areas or for established segments of Mission Areas. Since 
MA Strategies are designed both to relate the Program to 
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operational missions, and to facilitate understanding of that 
relationship by officials and organizations with influence 
over the Department of the Navy Program, MA Strategy Areas 
are structured to be compatible with the structure of Mission 
Areas used by DDR&E in the Mission Area Summaries (MAS) . 

Technology Area (TA) Technical Strategies . TA Technical 
Strategies set forth objectives for technology and the sup- 
porting rationales. The structure of areas for which TA 
Strategies are developed is also derived from the DDR&E mission 
Area Summaries. TA Technical Strategies so defined are also 
compatible with the Technology Coordinating Papers (TCPs) 
issued by the OSD. 

Preparation of Technical Strategies . Preparation of both 
MA and TA Technical Strategies involves the identification 
and evaluation of needs, and the identification of present, 
future, and potential capabilities for satisfying those needs. 
All Technical Strategy documents employ a similar format. 

Each describes the relevant Strategy Area (this will be either 
a Mission Area or a Technology Area) , sets forth Objectives 
and the supporting rationales, identifies how Objectives in 
the Strategy Area are being pursued currently, and references 
sources of additional information. 

Mission Area (MA) Technical Strategies . In general, each 
MA Technical Strategy sets forth: (1) Mission Area Objectives 

for the Mission Area, (2) the supporting rationales for those 
Objectives, and C3) a discussion of present and future 
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exploratory development efforts relevant to MA Objectives. 
Preparing MA Technical Strategies requires a comprehensive 
understanding of operational needs. 

Mission Need Information . Mission Need information is 
available through both formal and informal systems. Informa- 
tion on operational needs is provided primarily through an 
informal face-to-face system based on and backed up by a 
dociament-based system. Information on needs is derived from 
a variety of sources including Fleet letters, CNO memoranda, 
extrapolation of threat statements, etc. Mission Need infor- 
mation flows through the Navy/Marine Corps organization to 
the Exploratory Development Program by means of three formal 
docviments ; the Navy Science and Technology Objectives (STOs) , 
the Marine Corps General Operational Requirements (GORs) , and 
the Operational Requirements (ORs) . The first two documents 
are aimed primarily at guiding the inception of projects and 
programs (although guidance may be derived from them at any 
point in the process) . The Operational Requirements (ORs) are 
concerned with the transition of System Concepts to Advanced 
Development. 

Mission Area Objectives . MA Objectives are objectives 
for operational capabilities within the Strategy Areas. They 
are set forth as means-free statements of needs for opera- 
tional capabilities required to execute the mission. These 
statements are prepared in a manner which satisfies the 
requirements for statements of Mission Needs for the initiation 
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of major system acquisitions as set forth in 0MB Circular 
A-109 and implementing DOD directives and instructions. MA 
Objectives should be considered identical to the Mission 
Element Need Statements (MENS) defined in DOD Directive 
5000.1 and should conform to the format specified in Appendix 
D. All MA Objectives are to be nxambered in a manner which 
facilitates the information capabilities discussed in 4.6. 

The rationale for each MA Objective should be explained 
clearly and concisely and should reference all official docu- 
mentation (such as STOs and GORs) on which it is based in 
whole or in part. 

Actual and Potential Capabilities . Associated with each 
MA Objective is a discussion of means for satisfying the 
Objective. This discussion consists of a concise description 
of each system. System Concept, or other way (such as a combi- 
nation of forces) to satisfy the Objective, along with refer- 
ences to primary sources of additional information. Addition- 
ally, capabilities relevant to each Mission Need are to be 
identified and summarized briefly. These include present capa- 
bilities , systems in development or production but not yet 
operational, and Demonstrated- feasible, P resumed- feas ib le , 
and Pre-feasible System Concepts (SCs) . All SCs will include 
military worth ratings by CNO and/or CMC. The listing of 
Demonstrated-f easible SCs will also identify concepts which 
were dropped since the last update of the Technical Strategy. 
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Technology Area (TA) Technical Strategies. Each TA 



Technical Strategy: (1) Describes current technical efforts 

and relates these to identified Mission Needs, (2) sets forth 
Objectives for technology and the supporting rationales of 
those Objectives, (3) justifies the importance of technical 
opportunities and potential advancements in output-oriented 
terms, and (4) explicitly summarizes and presents justifica- 
tions for present, future and potential technological capabil- 
ities by specifically referencing relevant SCs. TA Technical 
Strategies are developed and/or updated in a manner analogous 
to the process described above for MA Technical Strategies. 

The process involves defining Technology Needs, setting forth 
the rationales for those needs, and describing present, future, 
and potential capabilities relevant to identified Objectives. 

A Technology Area is concisely defined in terms of the tech- 
nologies included within its scope. Where further clarity 
is needed, a Technology Area may be further defined by identi- 
fying related technologies beyond its scope. 

Technology Need Information . Technology Need information 
is derived from three principal sources; (1) the systematic 
analysis of Pre-feasible SCs, (2) feedback from the develop- 
ment and manufacturing processes, and (3) feedback from the 
operating forces . Information from analysis of SCs takes the 
form of documented statements of technological problems or 
barriers; feedback information flows primarily through person- 
to-person links and is introduced into the Strategy delibera- 
tions most frequently through the personal experience of 
Strategy Team members. 
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Technology Area Objectives . Objectives for technological 
capabilities are set forth as statements of needs for tech- 
nological capabilities, termed Technology Area Objectives, 
or TA Objectives. These are statements of needs both for the 
development of new technological capabilities in support of 
SCs , and for the preservation of a base of technology under- 
lying present operational capabilities (where it is judged 
that such a technology base would be endangered in the absence 
of action to preserve it) . TA Objectives are numerically 
identified in a manner which facilitates the information capa- 
bilities discussed above and in 4.6. The rationale for each 
TA Objective should be set forth clearly and concisely and 
should reference all official documentation (such as techno- 
logical problem statements from analyses of SCs) on which it 
is based. Every rationale should present a convincing case 
which describes how the Objective is critical either to 
achievement of higher level of mission capability or to the 
preservation of capabilities which would be endangered in the 
absence of actions the Objective is designed to stimulate. 

Actual and Potential Capabilities . Technological capa- 
bilities relevant to each TA Objective are identified and 
summarized briefly, as are the rationales for each of those 
capabilities. These include statements and rationales for 
current capabilities, for capabilities anticipated on the 
basis of planned technical effort, and for capabilities which 
are potentially achievable if a specified level of technical 
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effort is assumed. The accompanying discussion identifies 
the leading organizations possessing, acquiring, or capable 
of acquiring the capabilities. Relevant ongoing or planned 
technical programs are identified, and their predicted impacts 
on capabilities are specified. 

Program Plans . Program Plans are the primary instruments 
through which the DNT exerts top-down influence over portions 
of the Exploratory Development Program assigned to each SYS- 
COM and Laboratory. Program Plans document each SYSCOM's 
or Laboratory's integrated and balanced plan for maximizing 
the contribution of its own exploratory development efforts 
to Department of the Navy needs. Program Plans are not 
collections of Task Area Plans (TAPs) . (TAPs are prepared 
after the Program Plans are approved and are designed to 
carry out objectives listed in those Plans.) Analogous to a 
DCP for approval, funding, and management of an acquisition 
program, the Program Plan constitutes a "contract" between 
DNT and the SYSCOM 03/Lab TD for the Laboratory /SYS COM por- 
tions of the 6.2 Program. The Program Plan is the basis for 
the negotiation, approval, funding, and subsequent review 
of each SYSCOM's and Laboratory's Exploratory Development 
Program. The content and format for Program Plans are sum- 
marized briefly below. 

Technology Needs and Thrusts . This section will provide 
an introduction and overview of the proposed technical pro- 
gram, setting forth the major thrusts of the program and the 
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rationale for those thrusts. It will also discuss significant 
changes from past years and the rationale for those changes. 

Program Objectives . The technological objectives the 
technical program is designed to achieve will be set forth, 
as will the rationale for those objectives. These objectives 
are to be stated in quantitative terms (to the extent possible) 
to facilitate subsequent review of results. The rationale 
for objectives should explicitly identify the approved 
Technical Objectives which the program objectives are designed 
to support. 

Proposed Program . The technical program to achieve each 
program objective will be briefly described in both narrative 
and quantitative te 2 nns . This description should include the 
proposed dollars to be spent in support of each program objec- 
tive, the prospective performing institution (s) , and major 
milestones scheduled for achievement during the program year. 

In-House/Contract Ratio . The program plan will specify 
the relative proportions of work to be performed by Govern- 
ment organizations (in-house) and by industry and other non- 
Government organizations (under contract) . The DNT will 
negotiate the in-house/contract ratios with each SYSCOM and 
Laboratory to ensure that the overall Program conforms to the 
most recent policy guidance. 

Program Turnover . Ongoing efforts should be critically 
reviewed and sufficient work completed or terminated, to 
permit a minimum of 20% of the technical program to be composed 
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of new starts. The Program Plan will specify the percentage 
of the program v/hich is new, and will identify efforts being 
completed or terminated. 

Priorities ♦ The technical program will be organized into 
decision packages corresponding to discrete elements of the 
proposed program (such as all work related to specific System 
Concept or Laboratory subproject) . These decision packages 
will be presented, along with cost information, in a rank 
ordering. This is intended to facilitate the elimination, or 
addition, of entire projects (rather than to make across-the- 
board funding reductions which could result in s^±lcritical 
funding and the inefficient use of resources) . Rank ordering 
also facilitates the phased initiation of new starts to accom- 
modate Congressional budget actions and deferrals of funds. 

System Concepts . Priority rankings for SYSCOMs will be 
organized around the SCs which are the objectives of SYSCOM 
efforts. Information to be presented will include; 

a. A master rank-order of all ongoing and new starts of 
all three classes of SCs. 

b. A rank-order of both ongoing and new starts for each 
of the three classes of SCs. 

c. A rank-order for proposed new starts for each category 
of SC. 

Laboratory Subprojects . The proposed Laboratory program 
will be presented as a master rank-ordered list of subprojects 
and will include ongoing work and proposed new starts. 
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System Concepts (SCs ) . An SC is a System Concept for 
satisfying one or more MA Objectives. SCs provide a mechanism 
for; (1) identifying technology needs, (2) focusing technical 
effort on the technological advances required to make these 
high- leverage system concepts feasible, and (3) aiding trans- 
fer of that technology to development once technical feasi- 
bility has been achieved. Although normally originated by 
Laboratories, Systems Commands, or industry, SCs may be origi- 
nated by any organization. Instructions for the documentation 
of SCs are contained in Appendix E. 

Classes of System Concepts . SCs fall into three classes: (1) 
concepts presumed not yet technologically feasible without 
advances in underlying technology, (2) concepts presumed to 
be technologically feasible within five years but for which 
feasibility has yet to be demonstrated, and (3) concepts 
presimed ready for transition to development. Each of these 
three classes of SCs serves a different function in the man- 
agement of the Exploratory Development Program. All classes 
of SCs provide a basis for the assessment of a concept's 
military worth; this is an essential prerequisite to the fur- 
ther investment of technological resources (beyond the invest- 
ment required to define concepts sufficiently to permit 
meaningful military worth assessments) . 

Pre-Feasible System Concepts . Pre-feasible SCs provide a 
basis for identifying and defining technology which must be 
developed before these concepts become technologically feasible. 
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When communicated to the technical coinmunity involved in devel- 
oping broad-based technology, these technological require- 
ments stimulate development of the requisite technology. 
Pre-feasible SCs (and their related requirements for advance- 
ments in technology) are an essential link in the relevance 
audit trail; they make it possible to demonstrate the direct 
relevance of technical effort in the broad-based technology 
category to operational missions. 

Presximed-Feasible System Concepts . SCs for which feasi- 
bility has been presumed but not yet demonstrated provide 
the framework for; (1) defining the requirements for feasi- 
bility demonstrations, and (2) organizing related data. 
Demonstrating the feasibility of an SC can be likened to 
assembling a technological building block kit. As feasibil- 
ity is demonstrated for each essential technology, it is 
added to the kit until all essential building blocks are in 
hand. At that point the concept becomes a " Demons trated- 
feasible SC" and is ready for transfer to further development. 

Demons trated-Feasible System Concepts . Demonstrated- 
feasible SCs which are not immediately transferred to devel- 
opment may be dropped, or, they may be preserved as options 
for future development (depending upon the military worth 
assessment of the system user) . Where a decision has been 
made to preserve the option to develop, the SC provides the 
framework for the preservation of that option. The develop- 
ment option is preserved by continually: (1) monitoring 
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relevant technologies, (2) updating the information package 
to include current capabilities, and (3) identifying organi- 
zations which possess the needed capabilities. 

Assessment of the Military Worth of SCs . The CNO and CMC 
are invited to assess periodically the relative military 
worths of all SCs related to each MA objective in their areas 
of warfare. These military worth ratings, termed "V" ratings 
for value, are to be included in the discussion on MS Objec- 
tives in the Technical Strategy documents. 

Task Area Plans . Detailed plans for accomplishing the 
objectives in approved Program Plans are summarized in Task 
Area Plans (TAP) . Since program plan objectives are designed 
to support objectives in approved Technical Strategies, the 
Strategies provide amplifying guidance for the development 
of these more detailed plans . 

Information System . Because of the direct relationship 
between Technical Objectives called out in the Technical 
Strategies and objectives defined in the Laboratory and SYS- 
COM Program Plans, a relevance audit trail extends from each 
work unit to each assigned Department of the Navy mission. 

When the information system mandated by paragraph 4.b. of 
ASN (R&D) memo of 25 July 1975, "Management of the Exploratory 
Development Program,” is developed and in successful operation, 

it will be possible to make these relevance relationships 
visible through automatic data processing. An ADP query 
starting with a Mission Need will be able to identify all MA 
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Objectives called out in the MA Technical Strategy for that 
area, all Technology Area Objectives in support of MA Objec- 
tives called out in any of the TA Technical Strategies, all 
program objectives which support the MA Objectives either 
directly or indirectly through TA Objectives, and all work 
units supporting these program objectives. Going the other 
way, ADP capability based on the structure of objectives 
extending upward from the technical program, through the 
Technical Strategies to Department of the Navy missions, will 
make it possible to trace the impact of each work unit on 
higher-level objectives and Mission Needs. 



103 



i 



APPENDIX C 



STRUCTURE OF THE EXPLORATORY DEVELOPMENT PROGRAM 

The Chief of Naval Development (CND) establishes and 
promulgates the structural framework for planning, programming, 
and budgeting the Navy's Exploratory Development Program. This 
structure provides a link with DOD Program Elements, defines 
the scope of work within specific areas, and identifies the 
organization within the Navy responsible for planning, pro- 
gramming, and budgeting. 

Building Blocks of the Exploratory Development Program . 

The building blocks of the Exploratory Development Program 
are described in the following paragraphs. They are listed 
in descending order, from the largest block to the smallest. 

Program Elements . Program Elements are the smallest sub- 
division of the R&D program considered in the DOD programming 
system. A program is defined as an integrated activity, an 
identifiable military capability; a force, support activity, 
research activity, etc., comprising a combination of personnel, 
equipment, and facilities. In exploratory development, a 
Program Element consists of a number of projects in a related 
field within a single budget activity. 

Program Groups . Program groups are primary divisions 
within the Navy exploratory development planning system. They 
are used for management and data-processing purposes, and 
provide the links between the Navy planning system and the 
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Program Elements of the DOD system. They correspond one-to- 
one with the Program Elements — with the exception of Groups 
54 and 55, each of which corresponds to two Program Elements 
and thus requires the project area number to relate a compo- 
nent effort to the correct Program Element. The program 
groups are as follows : 



Program 
Group # 


Program Element Title 


Program 

Element 


11 


Undersea Target Surveillance 


62711N 


12 


Surface/Aerospace Target Surveillance 


62712N 


21 


Command and Control 


62721N 


31 


Missile Propulsion 


62331N 


32 


Strike Warfare Weaponry 


62332N 


33 


Undersea Warfare Weaponry 


62633N 


34 


Countermeasures 


62734N 


41 


Aircraft 


62441N 


42 


Nuclear Propulsion 


62542N 


43 


Ships , subs and boats 


62543N 


51 


Biomedical Technology 


62758N 
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Program 
Group # 


Program Element Title 


Program 
Element # 


52 


Ocean and Atmospheric Support 
Technology 


52759N 


53 


Logistics Technology 


62760N 


54 


Materials 


62761N 


54 


Electronic Devices 


62762N 


55 


Training and Human Engineering 


62757N 


55 


Personnel Support Technology 


62763N 


56 


C/B Weapons Defense 


62764N 


57 


Energy and Environmental Protection 


62765N 


61 


Laboratory lED 


62766N 



Functional Area . The functional areas are the five 
principal headings under which exploratory development is 
classified for program planning purposes. These functional 
areas are; 

100 - Target Surveillance 
200 - Command and Control 
300 - Weaponry 
400 - Naval Vehicles 
500 - Support Technology 

Project Areas . Project areas further subdivide the func- 
tional areas, as shown in Appendix and serve to group 
closely-related task areas. 



Proj ects . Projects encompass the aggregate of task areas 
under a given project area which fall within a single program 
group (therefore usually within a given Program Element) . 
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Siibproj ects . Subprojects encompass the aggregate of task 
areas within a project which are assigned to a single princi- 
pal development activity. Development activity identifying 



codes are as follows; 

Activity Code 

NAVAIRSYSCOM W 

NAVELEXSYSCOM X 

NAVFACENGCOM Y 

NAVMEDRSCHDEVCOM M 

ONR R 

NAVSEASYSCOM S 

NAVSUPSYSCOM T 

DLP Z 



Task Areas . Task areas encompass exploratory development 
effort directed toward a specific objective, e.g., thrust- 
controlled solid rocket. Task areas are the level at which 
detailed plans are prepared. These plans are available to 
the Chief of Naval Development as back-up data for justifica- 
tion of the Program to higher authorities, if needed. A task 
area may consist of one or more tasks and be assigned to an 
individual Laboratory, Systems Command, or Office of 
implementation . 

Tasks . Tasks are subdivisions of task areas established 
on the basis of organizations performing the work. They are 
used for internal Department of the Navy management only. 
Specific task assignments are made for the portion of the 
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work under a given task area to be contracted or to be 
assigned to an individual Laboratory or field activity. A 
task may consist of one or more work units. 

Work Units . Work units are the smallest segments into 
which research and technology efforts are normally divided 
for pxirposes of local administration. In the Exploratory 
Development Program, work units are subdivisions of a task 
selected by the organization performing the work to provide 
effective local technical control and supervision. A work 
unit is technically distinguishable in scope, objective, and/ 
or approach from other scientific or technical efforts with 
which it may be aggregated for financial or administrative 
purposes . 

Numbering System . Based upon the building blocks des- 
cribed above, the numbering system applied to specific devel- 
opment efforts within the Exploratory Development Program is 
designed to permit ready identification and retrieval of infor 



mation regarding any portion of the Program. An example of 
the use of the system is shown below: 

An Exploratory Development effort F 

in the "Missile Propulsion" Program Group .... 31 

in the "Weaponry"' Functional Area 300 

and specific Project Area "Solid Propulsion .... 332 

has Project Number F 31 332 

The portion performed by NAVAIR W 

has Subproject Number WF 31 332 
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A part of the effort is directed in the 

Task Area "Thrust Controlled Solid Rocket" 302 

and has Task Area Number WF 31 332 302 



Within the Task Area are Tasks / and within 
the Tasks are Work Units designated by 
three-character identifiers established 

respectively by the SYSCOMs and Labs xxx yyy 

So that the entire number identifying a 

Work Unit would be WF 31332302 xxx yyy 
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Work Unit Number 



Task Number 



Task Area Number 



Sub project Number 



Project Number 



W 

T 

Devel- 

oping 

Activity 

(NAVAIR) 



31 



Program 
(Explora- 
tory Devel- 
opment ) 



Program 

Group 

(Missile 

Propul- 

sion) 



*The three digit number 
and identifier assigned by 
SY S CCM /Lab or at ory 



332 



Func - 

tional 

■ Area 

_(We ap onry ) 

Project 
l-)Area (Solid 
Propulsion) 



302 * # 

1“ r~ 

Task Task 

Area 
(Thrust 
Control- 
led Solid 
Rocket 



// 



Work 

Unit 



110 



REFERENCE LIST 



1. Report of the Commission on Government Procurement: 

Part C - Acquisition of Major Systems ^ Vol. 2, pp. 55-65, 
U. S . Government Printing Office, December 1972. 

2. Office of Chief of Naval Development, Management Problems 

in Exploratory Development, supporting~paper for briefing 
for ASN (R&D) of 29 April 1975, 14 April 1975. , 

3. Jolly, J. A., Creighton, J. W., Technology Transfer in 
Research and Development , NPS-55Jo 75121, 1975 

4. McNallen, J. B. , Zand, D. E. , Lewin, A. Y., The Use of 
Models for Analyzing the Budget Decision Making Process , 
Armed Forces Comptroller, Volume 18 ( 2-4) . 

5. Caldwell, Dan, A Critique of Bureaucratic Politics , 
Stanford University, 1976. 

6. ASN (R&D) Memorandum, 25 July 1975, Management of the 
Exploratory Development Program . 

7. Booze, Hamilton, Allen, Review of Navy R&D Management 
1946-1973 , D.O.N. Publication, June 1976. 

8. General Dynamics-Pomona Division, Fiscal and Life Cycleg 
of Defense Systems , March 1976. 

9. Office of Director of Navy Technology, Status of 6.2 
Planning , Memorandum to MAT 035, 14 Jan 1977. 

10. Steiner, George A., Top Management Planning , pp. 657-692, 
McMillan Company, 1969. 

11. Hall, John T., Dixon, Roger A., Productivity Measurement 
in R&D , U. S. Department of Commerce/NBS Technical Note 
809, December 1975. 

12. Murphy, Richard, (Deputy Director Exploratory Development, 
Planning Office, NAVMAT 03T3) , personal interview, March 
1977. 

13. Unauthored article. House Unit Hits RDT&E Effectiveness , 
Aviation Week and Space Technology, p"^ 16 , 12 April 1977. 

14. Baker, N. R. , Pound, W. H., R&D Project Selection; Where 
We Stand , IEEE Transactions on Engineering Management, 
EM-11, pp. 124-135, December 1964. 

15. Trozzo, C. L. , Description and Critique of Quantitative 
Methods for the Allocation of Exploratory Development 
Resources , Institute for Defense Analysis , May 1972 . 



Ill 



0 




16. 



Albala, Americo, Stage Approach for the Evaluation and 
Selection of R&D Projects , IEEE Transactions on Engineering 
Management, November 1975. 

17. Ojdana, E. S., Weyent, J. P. , An Assessment of Selected 
Models Used for Evaluating Military R&D Projects , Rand 
Corporation, September 1976. 

18. Weiss, Carol H., Evaluating Action Programs; Readings in 
Social Action and Education , Allyn and Bacon, Inc., 1972. 

19. Weiss, Carol H., Evaluation Research - Methods of Assessing 
Program Effectiveness, Prentice-Hall, 1972 . 



112 



1 




BIBLIOGRAPHY 



1. Abernathy, W. J., Townsend, P. L. , Technology, Productivity 
and Process Change, Technological Forecasting and Social 
Change , 1~, 1975. 

2. ASN (R&D) , Formulation of Technical Strategies for the 
Exploratory~bevelopment Program" Keynote Address to East 
Coast Laboratories/Centers, 21 July 1976. 

3. ASN (R&D) , Improvement of the Exploratory Development 
Program , Draft memorandum to CNR, CND , DRDTE , DCS (RD&S) , 

21 April 1975. 

4. Augwood, D. R. , A New Approach to R&D Evaluation , IEEE 
Transactions on Engineering Management, Volume EM-22, 
February 1975. 

5. Augwood, D. R. , A Review of R&D Evaluation Methods , IEEE 
Transactions on Engineering Management, Vo lume EM- 2 0 , 
November 1974. 

6. Baker, N. R. , R&D Project Selection Models; An Assessment , 
IEEE Transactions on Engineering Management, Volume EM-21, 
November 1974. 

7. Buffa, E. S., Modern Production Management , Wiley and 
Sons, 1977. 

8 . Carroway , T . F . , Management of Navy Exploratory Develop - 
ment Program , Draft Memorandum to NAVMAT Code 03T3, 

6 October 1976. 

9 . Carroway , T . F . , The Navy Exploratory Development Program 
System , Draft of briefing outline presented to MAT 03T 
staff, 22 September 1976. 

10. Chief of Naval Development, Exploratory Development Program ; 
schedule for formulation of technical strategies , memor- 
andum for general distribution, 24 June 1976. 

11. Chief of Naval Development, Management of the Navy Explora - 
tory Development Program , NAVMAT Notice 3900, MAT 03T3, 

June 1976. 

12. Chief of Naval Material, Exploratory Development Program 

Management Manual, NAVMATINST 3900. X, draft of 31 January 

TTTT. 

13. Clarke, T. E. , Decision-Making in Technology Based 
Organizations , IEEE Transactions on Engineering Management, 
Volume, EM-21, February 1974. 



113 



14. Director of Navy Labotories, ROI for Technology Base 
Programs , personal letter to D. L. Potts, March 1977. 

15. Department of Defense Directive, Major System Acquisitions , 
DOD Number 5000.1, 18 January 1977. 

16. Department of Defense Directive, Major System Acquisition 
Process, DOD Number 5000.2, 18 January 1977. 

17. Early, Eugene H. , The Measurement of Efforts to Enhance 
the Effectiveness of NAVFAC RDT&E Investments , Master's 
Thesis, Naval Postgraduate School, June 1977. 

18. Gautschi, T. F. , An Investigation of the Management of 
Research and Development Projects , U. S. Naval Ordnance 
Test Station, Ad Pub 112, August 1962 . 

19. Lindblom, Charles E. , The Science of Muddling Through , 
Public Administration Review 18 , pp. 79-88, June 1959. 

20. McKenzie, R. V., Air Weaponry Block Study , Office of 
Management and Finance, Naval Weapons Center, Project 
082-174, September 1976. 

21. NAVSO, Department of the Navy RDT&E Management Guide , 

ASN (R&dI publication, NAVSO P-2457, 1 January 1975. 

22. Office of Management and Budget, Major System Acquisitions , 
Circular A-109, 5 April 1976. 

23. Rossi, P. H., Williams, W., Evaluating Social Programs ; 
Theory, Practice, and Politics, Seminar Press, New York, 
1972. 

24. Speelman, Ralph J. , Return on Investment, Residual Income 
and an Air Force R&D Laboratory , MBA Thesis, Ohio State 
University, 1971. 

25. Technology Plans Office, An Analysis of the Concept of 
"Strategy" and Its Application to Technical Strategies' 
in Navy Exploratory Development Management , An internal 
staff background document, 22 September 1975, (Amended 
August 1976) . 

26. Ulvey, A. P. , Ulvey, I. W., Analysis for Managers of 
People and Things , U . S . Government Printing Office. 



114 





i 




INITIAL DISTRIBUTION LIST 



No. 



1. Defense Doc\3mentation Center 
Cameron Station 
Alexandria, Virginia 22314 

2. Library, Code 0142 
Naval Postgraduate School 
Monterey, California 93940 

3. Library, Code 54 

Department of Operations Research 
and Administrative Sciences 
Naval Postgraduate School 
Monterey, California 93940 

4. Professor J. W. Creighton, Code 54Cf 

Department of Operations Research 

and Administrative Sciences 
Naval Postgraduate School 
Monterey, California 93940 

5. Dennis L. Potts, Code 3271 

Naval Weapons Center 

China Lake, California 93555 

6. Asst. Professor A. C. Crosby, Code 54Cw 
Naval Postgraduate School 

Monterey, California 93940 

7. Dean of Research, Code 012 
Naval Postgraduate School 
Monterey, CA 93940 



Copies 

2 

2 

1 



1 



1 

1 

1 



115 










I 



: V3305 



/ 



Thesis 
P7495 
c. 1 



Th^ 

P 



Potts 

An investigation of 
the effectiveness, 
selection and evalua- 
tion of navy explora- 
tory development pro- 
grams. 



Thesis 173)05 

P 7495 Potts 

c,l An investigation of 

the effectiveness, 
selection and evalua- 
tion of navy explora- 
tory development pro- 
grams. 



thesP7495 

An investigation of the effectiveness, s 




3 2768 001 92355 0 

DUDLEY KNOX LIBRARY 



